1072 Commits

Author SHA1 Message Date
Abin Simon
3b516e551f
Handle files moved multiple times within a delta query (#2560)
## Description

It is possible that within a single delta query that files could be moved multiple times, this fixes that.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2559

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-21 09:56:06 +00:00
Abin Simon
dd72c4c8f9
Mark collections as Deleted if not available and invalid delta (#2503)
## Description

<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Final entry : closes https://github.com/alcionai/corso/issues/2242

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-21 09:28:51 +00:00
ashmrtn
ed6335e590
New restore path for items with name in metadata file (#2495)
## Description

Add code to restore items that have their file name stored in the corso metadata file instead of as the name of the kopia file.

This code is not executed as the conditional for it will not trigger (comparison to max int)

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1535 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-21 04:26:25 +00:00
ashmrtn
4565c9f33b
Begin storing OneDrive file names in metadata (#2494)
## Description

This starts to store the name of the OneDrive file/directory in the corso metadata file for the OneDrive file/directory. This data is not yet used anywhere

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1535

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-21 03:12:16 +00:00
ashmrtn
cf62c6283c
Consolidate logic to see if Exchange/OneDrive user exists (#2508)
## Description

Use the check to see if a user has access to OneDrive/Exchange to also see if the user exists. This does not stop the population of GraphConnector users every time an instance is created, but does move us one step closer to that

Manually tested the following cases:
* backup "*"
* backup single existing user without error
* backup multiple existing users without error
* error when attempting to backup user that doesn't exist in the same domain
* error when attempting to backup user that doesn't exist in a different domain

Sample error output (progress disabled):
```text
No backups available
Error: 1 error occurred:
        * Failed to run Exchange backup for user foo@other.onmicrosoft.com: doing backup: producing backup data collections: resource owner [foo@other.onmicrosoft.com] not found within tenant
```

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1547 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-20 17:50:47 +00:00
Abin Simon
8096025c55
Print api response in logs for 4xx errors (#2544)
## Description

Print api response in logs for 4xx errors from GC. More info for things like 400 Bad Request could be useful.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-20 09:14:23 +00:00
Keepers
5707036b7c
add clues/fault to sharepoint api (#2507)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-18 20:42:48 +00:00
Keepers
b1ff20d36c
add clues/fault to sharepoint collections (#2506)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-18 19:51:51 +00:00
Abin Simon
7e3532832e
More fine grained error messages for retries (#2545)
## Description

This should help in figuring out more retry failures.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-18 17:16:42 +00:00
ashmrtn
e215462518
Add log messages about metadata saved/used (#2551)
## Description

Record the number of entries in the saved metadata. This could help catch odd situations like not finding anything to backup.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2550

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-18 06:25:05 +00:00
Keepers
207232e8d9
add clues/fault to sharepoint restores (#2505)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-17 22:05:32 +00:00
Vaibhav Kamra
daad056d7e
Pull in fix for concurrent map updates in clues (#2562)
## Description

Pulls in fix for a panic caused by concurrent map updates in the clues pkg

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-02-17 21:24:06 +00:00
Danny
62d652764a
GC: Ptr clean-up and Invalid NextLink logging (#2547)
## Description
- Use of ptr package for next link and delta links. 
- Added logging to verify the presence of receiving invalid next links from M365.


<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  No 

- Related to #2520
## Type of change
- [x] 🧹 Tech Debt/Cleanup

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x]  Unit test
2023-02-17 19:20:21 +00:00
Keepers
d395311821
error logging around malformed urls (#2561)
adds debug logging around url construction
2023-02-17 18:16:45 +00:00
Keepers
c92b70e000
wrap up clues and fault additions to exchange (#2504)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-17 17:08:08 +00:00
ashmrtn
0ff218ab3e
Hook into error callback in kopia (#2555)
## Description

Using this over extracting errors from the snapshot allows us to see all errors. The errors that each directory tracks are truncated when the directory is serialized for persistence if there're too many.

With the current `fault.Errors` behavior this cases the backup to fail if there's any errors
kopia ignored. This is consistent with behavior in the backup op that checks for any errors.

Unclear what the CLI output and log messages will be for these errors as they are grouped
into all other errors reported through the fault package

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2548

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-17 04:32:44 +00:00
ashmrtn
7b9ba935e8
log and continue if kopia can't find expected base snapshot directory (#2552)
## Description

This allows corso to continue making a snapshot even if the expected directory in the base is not found. The fact that the entry wasn't found will be logged

This should be safe so long as backups that had any sort of error, including "non-catastrophic" errors are marked as failed. Marking those as failed ensures that corso is using a matched set of metadata and data in the base snapshot rather than a partial snapshot

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2550

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-17 03:35:55 +00:00
ashmrtn
7995640109
Disable kopia optimization that lets us reuse a base directory (#2522)
## Description

Just handing the base directory to kopia doesn't let us see which items have been added which means we can't properly merge backup details later on.

Future implementations could do backup details merging based on path prefixes which could fix this problem.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2521

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-16 19:57:48 +00:00
Keepers
28ad304bb7
update items to accept ctx, fault.Errors (#2493)
## Description

In order for corso to track recoverable errors,
we need to pass a fault.Errors struct into the
items stream.  As long as we're doing that, we
might as well pass along the available ctx as well.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-16 19:09:20 +00:00
Abin Simon
55825afdbd
Deletion related fixes for OneDrive delta incrementals (#2518)
## Description

- parentReference for deleted items will not have path.
- GC was not responding with deleted state
- Deleted items are not streamed by kopia

Borrows some changes from https://github.com/alcionai/corso/pull/2503/files as that might be delayed with on merge. Will rebase that if this gets merged fist.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2117
* fixes https://github.com/alcionai/corso/issues/2517

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-16 04:19:50 +00:00
ashmrtn
476a931ccb
Update tests in kopia package (#2514)
## Description

Use the new suite.Suite wrappers in tester package. Also update suite.Run and some nolint directives.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2373

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-16 02:22:43 +00:00
ashmrtn
034ee0df39
Use suite wrapper in internal/common package (#2527)
## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2373

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-16 01:10:13 +00:00
ashmrtn
efe1b6292d
Extra test for some recursive delete behavior (#2478)
## Description

Folders replaced after a delete should not have their subfolders reparented to the new folder.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-15 21:14:32 +00:00
Keepers
7187b969e4
centralize observe channel listener handling (#2523)
## Description

To ensure that the observe channel handling
doesn't accidentally spawn unkillable routines,
adds a centralized channel listener func for
standard channel management in observe
progress bars.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-02-15 20:20:07 +00:00
Abin Simon
e60ae2351f
Set proper state and paths for moves in OneDrive (#2501)
## Description

Ties up the final piece of https://github.com/alcionai/corso/issues/2123. Add handling of moves for folder / files in delta response.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2123

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-15 18:27:45 +00:00
Keepers
26d286138b
add fault & clues to exchange api (#2492)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-15 17:01:04 +00:00
Danny
e70bf25f6e
GC: Restore: Libraries Workflow (#2511)
## Description
Due to recent changes with `Incrementals` restore of Library categories stopped working. This PR will allow for files that do not require special folder permissions to be restored. Changes involve the folder naming format that was coming from the CLI
<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes  #2489<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-02-15 13:30:03 +00:00
ashmrtn
28fcafe3b2
Consolidate/move OneDrive GraphConnector tests (#2487)
## Description

Now that we're also testing permissions behavior the OneDrive tests have gotten kind of large. This PR consolidates some of the code for tests and moves the OneDrive tests to their own file

It also sets things up so it's easier to add tests for different backup layouts in the future

Viewing by commit may make reviews easier

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2447

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-14 21:27:25 +00:00
Keepers
570ce85656
add clues/fault to exchange restore (#2491)
## Does this PR need a docs update or release note?

- [ ]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-14 19:33:07 +00:00
ashmrtn
31c9195a12
Proof of concept wrapper for suite.Suite (#2410)
## Description

Proof of concept for how to wrap suite.Suite and insert our own logic for determining if a test should be skipped or not.

Adds a new env var for running only integration tests

May require future work to add unit tests with the same (non-default) labels as their integration tests

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2373

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-14 17:56:26 +00:00
neha_gupta
2e128726c0
add retry middleware (#2406)
## Description

Added retry middleware which handles retrial of all HTTP request if,
- internalServerError 
- request timeout
 
with default no of retrials being 3.

## Does this PR need a docs update or release note?
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)
* https://github.com/alcionai/corso/issues/2287

## Test Plan
- [x] 💪 Manual
2023-02-14 22:58:16 +05:30
Abin Simon
f3c98c8fb7
Fix OneDrive permissions not getting backed up (#2473)
## Description

When we do a new backup, we look up the old permissions assuming they are accurate, but the permissions from the previous backup might be incorrect as we may have run it with permissions backup disabled.

It can also happen the other way round where even if we ask it to disable permissions backup, it can still end up having permissions which are pulled from the previous backup.

This fixes this behavior by setting the mod time for the backup info to the current time.

- [x] Add tests
- [x] Address TODOs in the PR

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2472

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-14 05:13:52 +00:00
Keepers
43e5ccdb5b
add fault/clues throughout exchange backup (#2382)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-14 01:30:55 +00:00
Keepers
52455356e5
add locationRef to details (#2427)
## Description

Adds a new reference to the details ent: location-
ref.  The location holds the human-readable
version of the item's location in whatever m365
service sourced the item.  Hookup is incomplete,
following PRs will fill out functionality.

Also adds a LocationPather interface to data_
collections to pass this data back and forth
between producers and consumers.

Should be safe to merge into main.

## Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

## Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix

## Issue(s)

* #2423

## Test Plan

- [x]  Unit test
2023-02-13 20:19:04 +00:00
Danny
67634cbbc1
CLI: Restore: SharePoint Pages (#2228)
## Description
Changes connect SharePoint Pages to restore CLI. 
<!-- Insert PR description-->

## Does this PR need a docs update or release note?
- [x]  No Arguments are hidden

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #2318 
* closes #2070 
* closes #2168<issue>
* closes #2218

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x]  Unit test
2023-02-13 18:48:33 +00:00
Keepers
89fb764526
add fault/clues to exchange data_collections (#2381)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-13 18:19:46 +00:00
Keepers
680ec2b751
adding fault/clues to non-service connector pkgs (#2380)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-13 16:54:09 +00:00
Keepers
f33dbc6351
deprecate adder interface for only fault.Errors (#2378)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-11 00:17:42 +00:00
Danny
eda4699d1a
GC: Use of Ptr extension (#2477)
## Description
Use of Pointer package in /connector/exchange

<!-- Insert PR description-->

## Does this PR need a docs update or release note?
Wanted to extend the package to generics, but it is incomplete

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related  #2474<issue>

## Test Plan

- [x]  Unit test
2023-02-10 21:10:22 +00:00
ashmrtn
6b6f684119
centralize folder creation and permission setting for OneDrive restore (#2468)
## Description

Rearrange how folder permissions are restored. Instead of restoring them when reading the meta file in the parent folder restore them when iterating through the collection itself.

This solution leans more heavily on the idea of having a map[folder path]->folder permissions and uses that to lookup what to restore

Folder permissions are still read and added to the map as they are encountered

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2447 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-10 20:22:43 +00:00
Abin Simon
b174c632b7
Handle deletions of collections in delta query (#2476)
## Description

Handle deletion items in delta graph API.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2123

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-10 18:56:40 +00:00
Abin Simon
2a72335716
Treat empty prev delta as invalid (#2475)
## Description

Empty delta values are treated as invalid and follow similar rules.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2123

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-10 18:25:21 +00:00
Abin Simon
66e5e7693d
Use ids for CollectionsMap for OneDrive (#2457)
## Description

This will help us with delta backups where we might run into a single
folder changing multiple times.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/pull/2407

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-10 17:57:10 +00:00
Danny
b3a1de89bb
GC: Item attachment contact support (#2465)
## Description
The logic for sanitizing Contactable data for restoration of `ItemAttachable.Contact` types. 

Contact `Item.Attachment`s required the removal of:
- `odata.Context` 
- `ETag` 
- `ParentFolder`

Otherwise, the following error occurs on POST. 
```bash
UnableToDeserializePostBody were unable to deserialize
```
<!-- Insert PR description-->

## Does this PR need a docs update or release note?
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2426<issue>

## Test Plan
- [x]  Unit test
2023-02-10 14:23:18 +00:00
ashmrtn
d9d0158b6f
Break file order dependency for OneDrive .meta files (#2450)
## Description

Begin using the Fetch() interface to retrieve OneDrive meta files inline when restoring the file. This removes the ordering dependency between .data and .meta files

This does not stop .meta files from being returned over the Items() channel. That can be disabled in a future PR

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2447

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-09 18:30:04 +00:00
Keepers
dad6776861
replace multierror with fault in setSites (#2377)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-09 18:11:59 +00:00
ashmrtn
754981d0d2
Make order insensitive permission compare (#2448)
## Description

Update the comparison for OneDrive permissions so it doesn't assume a fixed order. Also fix index out of bounds errors if backup did not retrieve the expected permissions

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2447 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-09 17:44:10 +00:00
Abin Simon
47d0eeb700
Move stateOf to data (#2455)
## Description

stateOf needs to be used in more places. Move it to a common location so  as to expose it.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*https://github.com/alcionai/corso/pull/2407

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-09 17:22:23 +00:00
Danny
2643fc2c89
GC: Backup: SharePoint: Pages Connect Pipeline (#2220)
## Description
Finalize the backup workflow for `SharePoint.Pages.`
Populate functions parallelizes
Fix for Incorrect Status during backup
<!-- Insert PR description-->

## Does this PR need a docs update or release note?
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes  #2071<issue>
* closes #2257
* related to #2173

## Test Plan
- [x]  Unit test
2023-02-09 14:39:00 +00:00
Abin Simon
81ecb072ff
Add make fmt for src (#2436)
## Description

Adds make command for code formatting.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
2023-02-09 05:25:01 +00:00