<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
Begins hiding PII with clues.Hide injections in our structured logging aggregators. This is a Non-
comprehensive change, and improvements here will be ongoing.
---
#### Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #2024
#### Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
in order to avoid errors cascading from graph API's
populated 204 responses containing `0/r/n/r/n`,
in particular in response to deletes, this change
hacks in the creation of a unique http client for
each delete call.
This is dangerous to do, since we already know
that generating clients per calls can leak
consumption of system resources such as sockets.
This change presumes that runtime deletes are
minimally, if ever, called during standard backup
and restore. The permission deletion in onedrive
is the stand-out exception that might need some
additional investigation.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #2707
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Updates and corrects input aliasing according to
the following rules (in priority order):
1. if the library name is usable, use it
2. if not, alias to the package name
3. if the package name is weird, alias sensibly
4. in case of collision, alias more distant imports
5. aliases should be consistent throughout
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.
Might follow up with a linter to enforce this change.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Moves the drive fetcher and pagers into the
onedrive/api package so that it can be easily re-
used by other packages. Also moves the mocks
out from local testing and into a mock package
nested in the api pkg.
first of a couple PRs fixing up sharepoint testing
issues.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] 💚 E2E
Few minor logic updates and remove a duplicate bit of code/test
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#2474
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
The tests only works once we have https://github.com/alcionai/corso/pull/2642 merged. I have tested it locally and confirmed that it works, but ask of now it is skipped. That said, I would like to get this merged if it is OK so as to not have to deal with merge conflicts later.
This as of now mimics incremental tests for Exchange in `TestBackup_Run_exchangeIncrementals`. Let me know if we should be having more.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2755
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
<!-- Insert PR description-->
Updates the call for `populateItems()` for OneDrive. Reduces the overall amount of calls made to obtain the Drive Name.
---
### Brief Description
- DriveName is saved with a collection field driveMap.
- The driveMap is a map where the key is the M365ID and the value is the display name for the drive
- During `populateItems()`, the driveName the map is sufficient for OneDrive but requires an update for SharePoint. During SharePoint process, the map is updated as additional driveLibraries are referenced.
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2732<issue>
#### Test Plan
- [x] ⚡ Unit test
Permissions are only stored if SharingMode is custom( "sharing" field is present). This will help with delta incrementals as well.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/2459
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
assuming it's possible for a graph
item to skip the malware detection,
we still want to catch and handle
400's from attempted malware
downloads downstream.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
#### Issue(s)
* #2701
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Graph exposes a "malware" filed in the delta response. This just logs that if we fail to download an item.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Renaming the funcs in the fault
package to be more clear about
their purpose and behavior. Largely
just find&replace changes, except
for fault.go and the fault examples.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Does this PR need a docs update or release note?
- [ ] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Changes:
- Use of `ptr` package to ensure safe dereferencing of pointers
- Update to query to ensure that `sharepointIds` are part of the response
- Alternate method created to obtain originating sharepoint site url
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1747<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
## Description
It is possible that within a single delta query that files could be moved multiple times, this fixes that.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2559
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Final entry : closes https://github.com/alcionai/corso/issues/2242
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
- Use of ptr package for next link and delta links.
- Added logging to verify the presence of receiving invalid next links from M365.
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
- Related to #2520
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] ⚡ Unit test
## Description
- parentReference for deleted items will not have path.
- GC was not responding with deleted state
- Deleted items are not streamed by kopia
Borrows some changes from https://github.com/alcionai/corso/pull/2503/files as that might be delayed with on merge. Will rebase that if this gets merged fist.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2117
* fixes https://github.com/alcionai/corso/issues/2517
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Added retry middleware which handles retrial of all HTTP request if,
- internalServerError
- request timeout
with default no of retrials being 3.
## Does this PR need a docs update or release note?
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* https://github.com/alcionai/corso/issues/2287
## Test Plan
- [x] 💪 Manual
## Description
Empty delta values are treated as invalid and follow similar rules.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2123
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Wire up configuring DoNotMerge for OneDrive collections.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2123
* https://github.com/alcionai/corso/issues/2124
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Forgot to push the review changes in https://github.com/alcionai/corso/pull/2371 .
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Pass the previous delta url fetched from metadata to collectItems and make sure we are using that when fetching the pages.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2123
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds more reties to handle timeout issues.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Create a pager for drive items that allows for better testing via mocking. Increased testing will come in later PRs
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2264
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Create a shared set of helper functions and interfaces for getting next and delta links from Graph API responses. This helps standardize how they are handled with respect to nil/empty values and makes comparisons easier in other code
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2264
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Make a separate interface for fetching drive information from Graph API. Interface allows for better testing via mocks
Merges SharePoint and OneDrive code for getting drives.
Also fixes potential bug where not all drives would be fetched. This could have occurred because the previous implementation for both SharePoint and OneDrive were not checking for paginated results
Viewing by commit is recommended
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2264
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
This check is brittle and shouldn't be needed. It should be sufficient to check whether a user has a
drive or not (similar to what we now do for exchange mailboxes).
Validated with the existing integration test.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2271
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Begin populating a global exclude set with files deleted from OneDrive. The set contains the IDs of files that have been deleted.
This is actually safe to return from GraphConnector now (though it's not returned) as it uses item IDs instead of item names.
Future PRs will need to address handling of (potentially) moved files
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2242
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Track folder moves and update the persisted metadata accordingly. Moves need to take into account subtree changes because OneDrive will not notify us of subfolders moving if nothing else was updated on the subfolder.
Handle updates by making a copy of the folder map for ease of updating and later on ease of finding out which collection paths have changed
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2120
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Follow up for https://github.com/alcionai/corso/pull/2156. We might deice to change where we add the logic to handle this, as in if we should move it to discovery module.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2145
* https://github.com/alcionai/corso/pull/2156
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Branch treats Drive `Site Pages` as a restricted directory. All `pages` within the directory, will be backed up via the Site Pages API. This adds an additional call to per collection to obtain the parent `driveName`.
<!-- Insert PR description-->
TODO: Create a Pipeline that distinguishes between SitePages, Libraries, and List for SharePoint
## Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
The documentation should state the assumption is that only `.aspx` files are to be found in the s
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2072<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Start persisting the folder path maps and delta URLs for backed up OneDrive/SharePoint drives. Delta URLs are saved in a map[drive ID]deltaURL while folder IDs are in a map[driveID]map[folder ID]folder path
Needs another patch to properly save the path for folders that match the selector, currently the selector comparison is only on the parent of an item
Later PRs can get the new folder map by taking the map from the previous backup and making changes to it when folder deletions/moves are encountered
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2120
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a few missing SKUs that license OneDrive. This is not the complete list.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2104
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds manual retry for drive retrieval <!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
## Issue(s)
* closes Issue #2044<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
This brings in another huge improvement in the backup speed dropping from ~15m to ~1.5m for an account with ~5000 files. This one also drastically reduces the number of requests we have to make for the same account from 5505 to just ~55. This also means that we don't get throttled anymore and we can easily run multiple backup jobs in parallel before we hit the 1024 limit.
~The code works as of now, but I have to double check the numbers as well as fix an issue with us opening too many files and causing program to crash with 'too many open files' when we bump up the numbers. With the current numbers, it works, but I wanna double check and optimize them. Plus some code cleanup.~
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Upstream Breaking changes applied for upgrade to the following libraries.
- kiota-abstractions-go from 0.14.0 to 0.15.1
- msgraph-sdk-go from 0.48.0 to 0.49.0
- msgraph-sdk-go-core from 0.30.1 to 0.31.0
- kiota-http-go from 0.9.0 to 0.10.0
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor --> Library support and Upstream bug fixes
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#1856 <issue>
## Test Plan
- [x] ⚡ Unit test
## Description
`graph.Service -> graph.Servicer`, no other changes.
More compliant with golang naming standards,
and will allow us to eventually migrate the
Service struct out of connector and into graph.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1725
## Description
Versioning upgraded for GC
Packages updated:
- [x] github.com/microsoftgraph/msgraph-sdk-go v0.41.0 => v0.48.0
- [x] github.com/microsoft/kiota-abstractions-go v0.14.0
- [x] github.com/microsoft/kiota-authentication-azure-go v0.5.0
- [x] github.com/microsoft/kiota-http-go v0.9.0
- [x] github.com/microsoft/kiota-serialization-json-go v0.7.2
- [x] github.com/microsoftgraph/msgraph-sdk-go-core v0.30.1
- [x] github.com/stretchr/testify v1.8.1
- [x] github.com/cjlapao/common-go v0.0.29 => v0.0.35
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
- Update library change. AdditionalData no longer contains `nextLink` for iteration. See commit [e7c75ca](e7c75ca1cc) for full details.
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1478<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Minor fixes for flakey tests:
* Dedupe OneDrive folders by checking IDs. Should help remove some test flakes.
* Weaken checks on kopia snapshot manager tests as they use maps with indeterminate iteration orders
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1660
* closes#1657
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Extends the site drive query to include a selector
for 'system' properties, which causes graph to
return an expanded set of drives including many
which are not normally visible to end users.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] 💚 E2E
## Description
Adapts the graph onedrive library to handle
access to drive data across both onedrive and
sharepoint services.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1506
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Corso should be able to have `Backup` called on a user that does not have OneDrive or does not have OneDrive files without producing an error. This feature branch creates OneDriveIntegration testing to be able to fetch OneDrive data for a particular user.
Logic changed within `drive.go`
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
**closes** #966
## Test Plan
- [x] ⚡ Unit test
## Description
migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.
⚠️ Your local environment will need to change in the following ways: ⚠️
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #558
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Hacks in the bytes written as the iteminfo size during onedrive item restores. Would use the drive itemable, but the size property isn't instantiated locally during creation, and cannot be mutated afterwards.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1113
## Test Plan
- [x] 💪 Manual
* Add functions for OneDrive purge
function is setup to return and take purgable instead of just userID,
itemID for deleter.
* Adjust other functions to take purgable
Adjust interface for deleter so that it allows more information to the
delete function. The called function can cast the purgable it was passed
back to the type that was returned by the getter function.
* Move struct definition
Per reviewer comment move struct definition to OneDrive package instead
of leaving in purge command.
* Fix lint error
Likely slipped through while GitHub actions was having issues.
## Description
Useful helper functions for the purge script
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1090
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Addresses a follow-up from the folder lookup implementation where we were not iterating through
all children (no pagination was implemented) when searching for a folder.
Instead of implementing pagination, this uses OneDrive path based addressing to do a direct lookup
for the folder.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #875
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
