510 Commits

Author SHA1 Message Date
ashmrtn
f71de7a021
Select specific metadata from base snapshot by Reasons base snapshot was picked (#1836)
## Description

Use the Reasons a snapshot was selected to retrieve only the metadata
corresponding to those reasons. This will avoid having multiple versions
of metadata for the same (resource owner, service, category) tuple as
well as pulling in more metadata than required for some backups.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1829 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-17 01:12:42 +00:00
ashmrtn
715e436dd9
Do not return an error if folder was deleted (#1849)
## Description

When backing up Exchange, don't return an error if the folder/calendar we're trying to fetch item IDs for has been deleted.

Error codes pulled with graph explorer API

Manually tested with an Exchange mail backup

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1846 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-17 00:06:44 +00:00
Keepers
6c72eefdae
deduplicate cache folders (#1814)
## Description

These two structures are effectively identical.
This refactor deferrs to the graph version since
services may still wrap that for more specific
behavior as needed.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1727

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-16 23:43:00 +00:00
ashmrtn
8c15c3ce16
Select subtrees of data from base snapshots based on reason snapshot was picked (#1828)
## Description

Pick what data to source from a base snapshot by examining the reason the snapshot was selected as a base. This helps avoid two issues:
* pulling in unwanted data when the base has a superset of what is being backed up. Example: pulling in contacts from the base when only email is being backed up
* clobbering already selected data from a different base with data in the base currently being examined. Example: two snapshots, one with contacts and emails and the other with just emails. Second snapshot is newer than the first. The email items in the first snapshot should not clobber those in the first when building the hierarchy

This PR also has the effect of, under some conditions, reducing the amount of data that is pulled from the remote kopia repo when building the hierarchy. This occurs because only the subtrees that will be used in the new backup are traversed instead of traversing the entire snapshot

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1740 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-16 22:31:13 +00:00
ashmrtn
da929d8448
Merge directory layout for kopia incremental backups (#1801)
## Description

Merge directory layouts between the passed in collections and the base snapshot(s). Also add unit tests to ensure the output kopia hierarchy looks as expected. (CLI) user observable behavior is not affected by this PR

This PR does not address:
* selecting subtrees for specific data categories in base snapshots
* not clobbering more recent info if multiple snapshots have subtrees for the same data category (ties into above)
* file deletions for services that can only report item deletions at a global level (e.x. OneDrive file deletions)

Viewing individual commits in PR may make changes easier to review

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1740

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-16 22:05:21 +00:00
Keepers
99b0f51980
consume prevPath on next backup (#1802)
## Description

Parses the previousPaths metadata collections
along with deltas, and hands paths down to
exchange backup collection producers.  Does
not yet scrutinize previous/current path diffs.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1726

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-16 19:59:34 +00:00
Vaibhav Kamra
3209cd1abf
Revert "temporarily skip e2e smoke tests" (#1843)
Reverts alcionai/corso#1835 now that #1841 is in. That should help with the flakiness.

I will follow up with another PR after though that moves these tests to a post-CI job (we don't need to run these on every commit)
2022-12-16 18:35:55 +00:00
Keepers
703acbdcf7
produce previousPaths metadata (#1799)
## Description

Adds an additional metadata collection: a folder
id to path string mapping.  This collection is
created on backup, and retrieved along with
the delta metadata on the next backup, but is
not yet parsed or utilzed downstream.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1726

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-16 18:09:25 +00:00
Danny
0d5043aa1f
SharePoint List selector Expansion (#1786)
## Description
Initial changes to support SharePoint Lists being chosen for Backup Operations. 
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

* close #1785<issue>

## Test Plan

- [x]  Unit test
2022-12-16 13:24:15 +00:00
Keepers
144d047568
temporarily skip e2e smoke tests (#1835)
## Description

These tests have the highes rate of timeout
based failures, while providing the most loose
gurantees about system behavior.  This will
temporarily disable them while we consider
separating our suite between integration
and e2e smoke sets.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Test Plan

- [x] 💚 E2E
2022-12-16 07:51:07 +00:00
Keepers
8d09612b81
correct backup tag key, metadata not found (#1830)
## Description

Fixes two bugs: 1. building the correc tag key
when retrieving the prior backup id from a
manifest.  2. returning ErrNotFound when a
snapshot entry isn't found for restore.  This
can occur for metadata if the service has
a prior backup that does not contain the
expected metadata files.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #1823

## Test Plan

- [x] 💪 Manual
2022-12-16 03:33:24 +00:00
Vaibhav Kamra
debe74a113
Add an updated flag to backup details (#1813)
## Description

Adds a flag in backup details that indicates whether the item in that backup is new/updated.

Currently always set to `true` but once we implement #1800 - we will set this to false for existing items from
base snapshots.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1812 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-15 04:19:35 +00:00
ashmrtn
0fbe65a304
Merge items in a single directory during snapshot (#1797)
## Description

Create and wire up a helper function that merges items in the base snapshot with items that have been seen from the collection

Will not change the output of execution at all since the base snapshot directory will always be nil at the moment

Will require further modifications to properly handle deleted items where only a single delta endpoint is used to fetch changes (e.x. OneDrive file deletions)

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1740

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-15 01:21:26 +00:00
ashmrtn
c1a0b74d22
Helper function to load snapshot root directories (#1796)
## Description

Add a helper function that allows loading the root directory of a snapshot given the snapshot manifest. This will allow future code for merging directory hierarchies to be tested without having to involve an actual instance of kopia with some previously created snapshots.

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1740

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-15 00:34:22 +00:00
ashmrtn
01b8111404
Say reasons why a base snapshot was selected (#1811)
## Description

For each snapshot manifest returned when looking for previous snapshots for a given set of owners cats, return the reason the snapshot was selected. This will allow other code to select the correct metadata and base snapshot subtree(s) when making incremental backups.

An example of when all metadata and directories in a base snapshot may not be needed is
```text
backup create exchange --data email,contacts --users user1 -> B1

// uses B1 as the base
backup create exchange --data email --users user1 -> B2

// uses B1 as the base for contacts and B2 as the base for email
backup create exchange --data email,contacts --users user1 -> B3
```

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1779

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-14 22:09:17 +00:00
Keepers
a10c1c6bbd
update time and status fields in op events (#1793)
## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1792

## Test Plan

- [x] 💪 Manual
2022-12-14 17:43:51 +00:00
ashmrtn
12544d88d3
Have BackupOp pass in OwnersCats to kopia.BackupCollections (#1805)
## Description

Instead of relying on KopiaWrapper to create the OwnersCats for a
backup, have BackupOp create them from the selector and pass them in.

This is necessary as incremental backups will no longer see all the data
in the backup, meaning it cannot accurately create the OwnersCats
because some data categories or owners in the backup may not have had
changes.

OwnersCats are eventually converted to tags on a kopia snapshot and used
to lookup snapshots when trying to find base snapshots for incrementals.

Additional minor changes:
* use pointers instead of values when passing parameters
* set backup details OwnersCats to nil

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1781  

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-14 16:36:39 +00:00
Keepers
8a29c52cdc
move graphService to graph as Service (#1790)
## Description

Relocates the graphService struct to graph as
the Service struct.  Replaces GC's embedded
graphService with a graph.Servicer reference.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-14 01:39:00 +00:00
Keepers
e2775aeb95
Refactor service failfast (#1789)
## Description

Configuration and attenion to the graphService
failFast is haphazard and has shared ownership.
This change removes that property from the
service, along with the ErrPolicy func, in favor of passing around a control.Options struct.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725
* #302

## Test Plan

- [x]  Unit test
2022-12-13 23:06:27 +00:00
ashmrtn
865edac9bd
Update to go1.19 for corso (#1632)
## Description

Required as some third party libraries are now requiring 1.19.

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #1629 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-13 21:01:55 +00:00
ashmrtn
8442175a5d
Allow updating and returning non-default values (#1794)
## Description

Update mockcollection to allow returning non-default values for attributes like PreviousPath() and whether an item has been deleted.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1740 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-13 20:41:53 +00:00
Keepers
414d2a490f
consume old deltas on exchange mail backup (#1768)
## Description

When backing up exchange data, parse the
metadata collection of delta urls from prior runs
(if any exist) and pass those tokens along to the
fetch functions for re-use.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
2022-12-13 19:03:33 +00:00
Danny
d8172de67e
GC: BackUp: SharePoint Test Updates (#1783)
## Description
Addition of SharePoint-specific functions to test package. Verifies that support package functions are present and do not cause errors upon SharePoint Collection Creation 
## Type of change

- [x] 🐹 Trivial/Minor

## Test Plan
- [x]  Unit test
2022-12-13 18:27:41 +00:00
Keepers
fc5f42545f
rename graph.Service to Servicer (#1787)
## Description

`graph.Service -> graph.Servicer`,  no other changes.

More compliant with golang naming standards,
and will allow us to eventually migrate the
Service struct out of connector and into graph.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725
2022-12-13 17:37:03 +00:00
Keepers
4d46847f6c
retrieve manifests and metadata from kopia (#1744)
## Description

In a backup operation, begins the operation by
retrieving all backup manifests and metadata
from prior operations.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-12-12 22:36:50 +00:00
Keepers
565c33af4e
generate a unique repository id on init (#1716)
## Description

generates a unique repository ID on repo init, and retrieves it again on repo connect.  This allows
the event bus (and any other tracker) to id a repo independent of other context.

I've been unable to find any good way to retrieve a unique repository ID from kopia.  This seems like
the next quickest solution.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #1602

## Test Plan

- [x] 💪 Manual
2022-12-12 22:14:37 +00:00
ashmrtn
893bc978ba
Use io.MultiReader to inject kopia file version (#1767)
## Description

Instead of rolling our own logic for injecting a version, use a
MultiReader to concatenate the streams. Handling Close() is now more
complex though.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #1766 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-12 21:46:38 +00:00
Danny
dd96a87611
GC: SharePoint: Helper function consolidation. (#1782)
## Description
Move helper functions & test structs to `helper_test.go`
<!-- Insert PR description-->

## Type of change

- [x] 🐹 Trivial/Minor


## Test Plan

- [x]  Unit test
2022-12-12 20:14:37 +00:00
Abin Simon
1d16355873
Don't retry unless timeout err (#1738)
## Description

We have a retry handler within the client added as a middleware. So, don't retry here again if we still get any other err.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/1684

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-12-12 08:20:02 +00:00
Vaibhav Kamra
93a7ff27e3
Store backup details in a separate snapshot (#1755)
## Description

This PR uses a separate snapshot to store backup details instead of a Corso model (i.e. a kopia manifest).

Introduces a `StreamStore` that can be leveraged to store larger metadata objects. We can also leverage this
for incrementals or restartable backups going forward.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1735 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-10 00:47:49 +00:00
ashmrtn
be9b214c0c
Track more info about collections during backup (#1765)
## Description

Track additional information about collections and their items during backup so we can properly merge directories and items in directories when doing incremental backups

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1740 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-09 23:55:45 +00:00
Georgi Matev
21c717fbe8
Rename events to adhere to the "Noun Action" naming convention (#1763)
## Description

Rename events to adhere to naming convention 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-09 22:47:51 +00:00
ashmrtn
e1738d657d
Shuffle around some kopia snapshot code (#1764)
## Description

Pull code specific to making snapshots in kopia into a separate file. Factor out code specific to handling collections in preparation for having to deal with kopia items as well during incremental backups. Apart from code movement and factoring into functions no other changes have been made

Viewing by commit will make changes easier to see

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1740 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-09 21:58:16 +00:00
Keepers
8b2489b60c
export makeTagKV, fetchPrevSnapshotManifest (#1750)
## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725
2022-12-09 19:34:29 +00:00
Keepers
edd0708285
aggregate categories from selector (#1742)
## Description

Adds a PathCategories() func to selectors which returns
all the path categories exhibited by the scopes within the
selector.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
2022-12-09 18:57:06 +00:00
ashmrtn
2524523298
Add a comment for updating Schema constants (#1759)
## Description

Note what things need to stay the same and what things can change when updating schema constant values. This is required because some information about the constants are persisted in the model store.

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #1757 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-09 18:00:45 +00:00
Abin Simon
4ee66c3d3d
Refactor http client creation for onedrive file download (#1736)
## Description

This way we are able to use any middlewares added in both places.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-12-09 04:59:17 +00:00
Keepers
ca5f197067
export ownerCats and fetchPrevSnap (#1741)
## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725

## Test Plan

- [x]  Unit test
2022-12-09 01:34:20 +00:00
Keepers
38239e986c
add weburl identification in the CLI (#1712)
## Description

Adds weburl identificatiotn and unioning in the
cli, for both backup and restore of sharepoint
data.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1616

## Test Plan

- [x]  Unit test
2022-12-09 01:14:04 +00:00
Danny
45e535c9b8
GC: Go.Mod update (#1479)
## Description
Versioning upgraded for GC
Packages updated:
- [x] github.com/microsoftgraph/msgraph-sdk-go v0.41.0 => v0.48.0
- [x] github.com/microsoft/kiota-abstractions-go v0.14.0
- [x] github.com/microsoft/kiota-authentication-azure-go v0.5.0
- [x] github.com/microsoft/kiota-http-go v0.9.0
- [x] github.com/microsoft/kiota-serialization-json-go v0.7.2
- [x] github.com/microsoftgraph/msgraph-sdk-go-core v0.30.1
- [x] github.com/stretchr/testify v1.8.1
- [x] github.com/cjlapao/common-go v0.0.29 => v0.0.35

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
- Update library change. AdditionalData no longer contains `nextLink` for iteration.  See commit [e7c75ca](e7c75ca1cc) for full details. 

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1478<issue>

## Test Plan

- [x]  Unit test
2022-12-08 21:54:22 +00:00
Keepers
53195017a1
minor refactor for cleanup in operations (#1729)
## Description

Starting off changes with a minor refactor.  Hopefully will
keep code cleaner as we populate the backup operation in
the coming changes.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1725
2022-12-08 21:33:53 +00:00
ashmrtn
19f7de59f4
Expand interfaces for components used during backup (#1731)
## Description

Expand interfaces for `GraphConnector.DataCollections` and `kopia.Wrapper.BackupCollections` to include parameters that will be needed during incremental backups. This patch only expands the interfaces, it does not add any extra functionality and the passed parameters are currently ignored.

In the future, passing nil for any of the new parameters should result in the current "full backup" behavior that Corso has. Passing values in these parameters should enable delta token-based incremental backups (assuming all the required data is there for the incremental backup)

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #1700 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-08 18:40:10 +00:00
ashmrtn
e15d86e82f
Persist Exchange delta tokens (#1722)
## Description

Begin persisting Exchange delta tokens for data categories that support delta queries. Tokens are stored in a single file as a `map[M365 container ID]token` where both the container ID and token are of type `string`. The file is located in the kopia snapshot that has all the other backup data at the path `tenant-id/{service}Metadata/user/category/delta`. No information about the delta token file is stored in backup details.

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1685 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-08 18:01:00 +00:00
Keepers
4c976298d4
populate sharepoint itemInfo (#1687)
## Description

Currently, all drive backup and restore actions
populatet a details.OneDriveInfo struct.  This
change branches that struct between one-
drive and sharepoint info, depending on the
current source.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1616

## Test Plan

- [x]  Unit test
2022-12-08 17:26:56 +00:00
Keepers
9cab212776
amend test user cleanup list (#1724)
## Type of change

- [x] 💻 CI/Deployment

## Test Plan

- [x] 💚 E2E
2022-12-08 16:58:32 +00:00
Keepers
a6aa86ce5c
add a weburl to siteid reducer in gc (#1671)
## Description

Adds a func in graphConnector that reduces
siteIDs and webURLs into a set of siteIDs.  This
will be used by callers such as the CLI to
generate id-based selectors for sites even if
they handle webURLs as an alternative id.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1616

## Test Plan

- [x]  Unit test
2022-12-08 15:38:24 +00:00
Abin Simon
6ee121a70a
Use corso version with event reporting (#1721)
## Description

Add version of the current binary when reporting.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1594

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-08 08:07:52 +00:00
ashmrtn
0ad6f440ea
Disable kopia-assisted incrementals for OneDrive (#1728)
## Description

Do not implement the StreamModTime interface so kopia-assisted incrementals cannot be used for OneDrive.

Feature should be reenabled when the following are resolved:
* #1702 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #1723 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-07 23:07:35 +00:00
ashmrtn
09c48c1ec9
Expand Stream and Collection interfaces for delta token-based incrementals (#1710)
## Description

Add functions to Collection and Stream interfaces that allow for getting
more information about the difference between the previous backup and
the currently in-progress one. These will allow delta token-based
incremental backups to determine how the state has evolved.

Current code does not use these functions and return values for them
are "default" values that should result in full backups even if
KopiaWrapper is updated to start checking the values and GraphConnector
still pulls all items

These functions are not used during restore and can return "default"
values

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1700 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-07 22:20:59 +00:00
ashmrtn
14a3c2e189
Create in-memory MetadataCollection type (#1719)
## Description

Adds an in-memory collection type that can be used to pass metadata files from GraphConnector to KopiaWrapper. Meant for only small amounts of data as everything must be buffered in-memory in the collection

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1685 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-07 17:07:56 +00:00