977 Commits

Author SHA1 Message Date
ashmrtn
c4e9046870
Fix wsl lint errors in kopia package (#650) 2022-08-26 08:13:25 -07:00
Keepers
f1370b36e4
add cli commands to delete backups (#641) 2022-08-24 18:26:14 +00:00
Keepers
db2c1ec8e2
delete backups in modelStore and snapshot (#640)
Introduces manual deletion of existing backups.  The delete
includes: the modelStore backup, modelStore details, and
the kopia snapshot of the backup itself.
2022-08-24 16:30:27 +00:00
Keepers
15b12e634d
name cleanup in selectors and onedrive (#643) 2022-08-24 15:13:04 +00:00
Keepers
2d88e59cd0
Issue-556 introduce discreteScopes in selector handling (#635)
Graph may have an easier time parsing scopes if the root
user is discretely identified, as opposed to being a wildcard.
DiscreteScopes() automatically handles replacement of
scopes matching Any() user with the set of user provided
discrete IDs.
2022-08-23 21:56:24 +00:00
Danny
3b9c8e284c
GC: Reduce the amount of folders upon restore (#637)
* Issue #595: All collections with the same `fullPath` name are to be stored within the same folder.
2022-08-23 20:46:59 +00:00
Danny
6e7335cc4d
Gc restore multi framework (#639)
* Abstraction extended to restore. Application logic moved to service_functions.go
2022-08-23 20:30:46 +00:00
Keepers
b524053e96
resolve lint errors (#638) 2022-08-23 20:22:25 +00:00
Danny
a1f79862f1
GC: Event collections to be referenced by calendarID instead of folder (#636)
`data.Collection` of type event  `fullPath()` contain the calendar id
2022-08-23 17:38:30 +00:00
Keepers
99908b914d
propagate test cache dir in cli integration tests (#634) 2022-08-23 17:13:07 +00:00
ashmrtn
e8e4bf0914
WIP 316 kopia upload errors (#330)
* Update to kopia with required callback

* Support structs for materializing backup details

Kopia will not allow us to pass data to it that should be passed back to
us in the `FinishedFile` callback. To work around this, create a small
thread-safe support struct that handles information about files kopia is
currently processing. Entries are removed from the set when kopia is
done with them and if no error occurred, the item's info will be added
to the BackupDetails.

* Switch to best attempt for iterating through files

Defaulting to "best-attempt" error handling where all data that didn't
result in an error is handed to kopia and then all errors encountered
are returned at the end.

* Test for uploads that have an error

Simple error reading a file. BackupDetails should not contain
information about the file that had the error (needs update to kopia
code and this code to pass). All other files should be present in kopia
and in BackupDetails.

Co-authored-by: Danny <danny@alcion.ai>
2022-08-23 08:18:29 -07:00
Danny
eca16e4a69
CLI: Contact and Event Flags active (#633)
Contact and Event flags restored
2022-08-23 12:29:19 +00:00
Keepers
8d2a437f1d
clean up scope production, cat comparators (#630)
Scope production was still using service-type specific
factories instead of a generic factory set.  This has been
centralized so that all service instances share the same
scope production concerns.

Additionally, category comparator funcs now use generic
comparators as well, which allows for the removal of the
isType() and includesType() comparator funcs.
2022-08-22 18:02:54 -06:00
Keepers
e7b863c444
onedrive selector interface compliance (#626)
Scope filtering is currently hardcoded to the exchange
use case.  In order for future work to rely on boilerplate
rather than re-writing the full filtering logic on each new
type, as much of that code as is possible has been moved
into a generic toolset.
2022-08-22 21:35:46 +00:00
Danny
d7abed1406
GC: setTenants not checking *users.GetMail() (#629)
User query updated from `*user.GetMail()` -> `*user.GetPrincipalName()`. Guard checks added and added GetUsers as one of the GraphQueryFunctions
2022-08-22 14:07:57 -07:00
Danny
b573882746
GC: Accept selectors.BackupExchange exchange.Event and exchange.Contacts (#605)
GraphConnector suite expanded to support backup of Mail, Contacts, and Events. e2e testing suite expanded for all 3 applications.
2022-08-22 20:52:01 +00:00
Keepers
d954c68216
optional arm docker image build (#624)
adds the `--arm` flag to /corso/build/build-container.sh
2022-08-22 19:22:09 +00:00
Keepers
12dbfce6d6
refactor scope filtering (#555)
Scope filtering is currently hardcoded to the exchange
use case.  In order for future work to rely on boilerplate
rather than re-writing the full filtering logic on each new
type, as much of that code as is possible has been moved
into a generic toolset.
2022-08-22 19:08:14 +00:00
ashmrtn
7d057dd2ac
Update to kopia with StreamingFile caching (#617) 2022-08-22 15:31:12 +00:00
Keepers
b483db0228
make exchangeScope comply with scoper (#619)
Centralizes as many of the exchange scope funcs as
possible into scopes.go.  Ensures exchangeScopes comply
with the scoper interface.  Reshuffles some test helper
code in selectors to a centralized file.
2022-08-19 22:46:55 +00:00
Keepers
ade400126d
final cleanup for new PR (#609) 2022-08-19 20:40:56 +00:00
ashmrtn
35713acffb
Regression test for kopia sharing local config (#620)
Ensure that kopia is not sharing the local config and that different
instances of kopia reconnect to the proper remote repo.
2022-08-19 20:33:07 +00:00
Danny
4f5bb2b995
Exchange event info (#569)
Exchange Info feature added and verified with `mockdata` to ensure correctness
2022-08-19 13:43:26 -06:00
Keepers
daa939f5f8
fullfill categorizer for exchange selectors (#602) 2022-08-19 17:43:53 +00:00
Niraj Tolia
4bf6e92098
Docs text and build fixes (#613)
* Fix Lint warnings in the linters docs file

* Fix broken links (`make build` is no longer broken)
2022-08-19 09:46:24 -07:00
Niraj Tolia
9755b7a72b
Docs upgrades (#596) 2022-08-19 16:15:14 +00:00
Danny
3e1a060f6a
Upgrade msgraph-sdk-go version from 28 to 34 (#612)
Upgrade graph version from 28 to 34. No changes to source code
2022-08-19 15:51:00 +00:00
Danny
74a8666956
Exchange Serialization test finalized (#611)
Finalize graph connector regression tests. Needed to change the use of GC to be singular due to parallel testing.
2022-08-19 15:39:16 +00:00
Danny
9d18d50cf8
Populate function Refactor to single function (#585)
Populate functions streamlined into one function with a few additional abstractions
2022-08-19 08:19:42 -07:00
Keepers
854635ac24
allow tests to set a specific kopia config dir (#607) 2022-08-19 00:24:32 +00:00
Vaibhav Kamra
c3c361282d
Onedrive selectors (#594)
## Description

Introduces OneDrive selectors - currently only users are supported.

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
- #388  

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-08-19 00:00:28 +00:00
Vaibhav Kamra
8fd867745f
Onedrive item reader (#582)
## Description

This contains the following changes:
- Support functions to enumerate a users drives and the files within it.
- `driveItemReader` method that that reads a drive item
- Integration tests for the above

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
- #388 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-08-18 23:38:23 +00:00
Keepers
649255f112
fix suite in restore cli integration tests (#603) 2022-08-18 22:52:33 +00:00
Keepers
42df5cef58
push doc autogeneration changes to new branch (#531) 2022-08-18 22:34:03 +00:00
Vaibhav Kamra
350b148155
Instructions on running tests (#588)
* Add testing instructions

* Update link
2022-08-18 20:09:40 +00:00
Keepers
91e5c3d90c
add the corso env command for env var guidance (#591) 2022-08-18 19:09:25 +00:00
Keepers
3ac05acf64
introduce log-level control (#589)
Introduces the log-level flag, defaulting to info.  Also does
a minor refactor of how Print is called for backup results,
which moves the backup/details imports out of the cli/print,
and instead has thoses packages call a Print func.
2022-08-18 18:30:15 +00:00
Keepers
c13ef798eb
use scope type instead of map (#597) 2022-08-18 18:06:22 +00:00
Keepers
e198aa9ae7
add scopes.go (#579)
Introduces selectors/scopes.go, which is the base file for
managing service-agnostic scope logic.  Funcs like "isAny",
"contains", "getValue" and etc are expected to be housed
here, so that service instances only need to provide thin
wrappers, mostly typecasting, around their behavior.
2022-08-18 11:21:14 -06:00
Vaibhav Kamra
f8bfe93261
Scope restore CLI test (#592) 2022-08-18 14:19:08 +00:00
Georgi Matev
0ebf4d33e3
Start building out README and basic structure (#590) 2022-08-18 13:42:26 +00:00
Danny
ce087a2fd2
Fixes the resource-not-found-fix (#587)
Bug fix-> UserID replaces the use of a TenantID on query.
2022-08-17 18:17:34 -04:00
Vaibhav Kamra
c4e5915983
OneDrive Collection (#580)
## Description

Introduces a OneDrive data collection.

Follow up PRs will implement the `collection.driveItemReader()` method that uses the Graph API

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
- #387 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-08-17 19:38:05 +00:00
Keepers
a1b9f876ae
add final basic cli tests (#578)
Adds the base CLI integration tests for backup, list, details,
and restore.  Also refactors out the global root command
value in favor of a ctx-bound reference so that tests may
control safely overwriting stdout to scrutinize output.
2022-08-17 18:13:55 +00:00
Danny
9049f3c2bf
552 gc mockdata events (#583)
Addition of mock event data structure and testing suite.
2022-08-17 11:22:39 -06:00
Danny
90be4200ca
GC: Events Support added. Not connected within ExchangeDataCollection (#516)
m365 events added for querying and iteration. Test suite added. Mock data is in a different PR.
2022-08-17 16:29:37 +00:00
Danny
ad2c17876f
ContactInfo added to exchange package (#574)
ContactInfo created for models.Contactable. ContactInfo test suite created.
2022-08-17 13:47:27 +00:00
Danny
41f22ad9a4
Added error trace reading for query calls. (#575)
Added error tracing that gives more details for failed GraphQuery reading for query calls.
2022-08-17 13:07:53 +00:00
ashmrtn
24b40cf601
Persist and enable kopia compression options (#577)
* Skip updating policy if no changes needed

* Set default compressor on repo connection and initialization

* Factor out common code for connecting to a repo

* New unit test ensuring compressor is set properly

Test compressor is set both through Initialize and Connect because
client code doesn't have to call Connect if it just did Initialize.
2022-08-16 23:19:57 +00:00
Vaibhav Kamra
f870667856
Fix docker build command (#521) 2022-08-15 23:16:43 +00:00