Minor cleanup that will also help reduce diff for future changes.
Instead of taking in a details builder and adding to it during
restore, just create a local details builder and return the built
details to the caller
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4254
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Add logic and tests for preview backups in drive-backed services.
Does slightly change a few of the options for preview backup limits
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Switch to a cleaner way of disabling merge bases when making a preview backup. This sets the flag to not find merge bases instead of finding them and then dropping them.
Manually tested running a backup with `--preview` (local dev flag) and
`--disable-incrementals` and verified in the log that merge bases were
dropped
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
three changes
1. bumps the xsync package to v3
2. creates a common package for synced maps
3. replaces all xsync MapOf imports with the new common/syncd package.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Just shuffling some variables and log statements around. No logic or
test changes beyond that
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
We were missing a retry for connection timeout issues. Adding it.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
This completes the wiring for Exchange preview backups. A few notable
points:
* Adds new config substruct(?) for limits on items/data
* Adds "reasonable defaults" (very small) for Exchange data categories
* Adds "important" and "skip" containers for Exchange data categories
All the above can be tweaked as we determine what values work best
Manually tested
1. regular backup
2. preview backup
3. regular backup
Verified that no merge base is used for the preview backup and the
merge base made in (1) is used for the backup in (3)
This feature is not exposed via CLI
May be easiest to review by commit
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issues
Merge after:
* #4657
* #4607
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
sdk consumers need a way to configure the graph api client options when using the services api. This is the first in a two-part change to expose those options. This step moves the api to a client-based set of methods instead of free functions. The next step will add client configuration to the service client.
---
#### Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
#### Type of change
- [x] 🌻 Feature
#### Test Plan
- [x] 💚 E2E
A handful of the error sentinel values in graph/errors.go were only around to comply with an old expectation that IsErr funcs have a matching sentinel. That assumption is out of date, and there's no point in keeping around unused sentinels.
This PR makes three changes to clean up that issue:
1. remove unused sentinels (unused is qualifed as "not produced").
2. fix external pacakges which referenced those errors in tests alone by replacing those sentinels with odata errors.
3. centralize odata error test code to help support upstream package odata error creation.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Increase proliferation of the count bus to record runtime stats.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Just a few minor updates to stats logic to reset the counters if the delta is reset and also only count items if they weren't previously seen and counted.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Previously we were printing a table for a single item which, for one
was unnecessary, but this also make the output of a backup more
compact.
Before:
```
Logging to file: /home/meain/.cache/corso/logs/2023-10-20T10-21-23Z.log
Connecting to M365 done
Connecting to repository 1s done
Backing Up ∙ Teams Testing (meain)
Discovering items to backup 27s done
Libraries (TeamsTestingmeain) done (found 0 items)
Libraries (TeamsTestingmeain-Shared0) done (found 0 items)
Libraries (TeamsTestingmeain-Private2) done (found 0 items)
Messages 7s done (found 6 channels)
Backing up data 7s done
Backup complete ∙ 7b40dd40-f808-4d57-8e39-b4553e48dc5d
ID Bytes Uploaded Items Uploaded Items Skipped Errors
7b40dd40-f808-4d57-8e39-b4553e48dc5d 0 B 0 0 0
Completed Backups:
ID Started At Duration Status Resource Owner
7b40dd40-f808-4d57-8e39-b4553e48dc5d 2023-10-20T10:21:32Z 36.632747912s Completed Teams Testing (meain)
```
After:
```
Connecting to M365 done
Connecting to repository 1s done
Backing Up ∙ Teams Testing (meain)
Discovering items to backup 31s done
Libraries (TeamsTestingmeain) done (found 0 items)
Libraries (TeamsTestingmeain-Shared0) done (found 0 items)
Libraries (TeamsTestingmeain-Private2) done (found 0 items)
Messages 9s done (found 6 channels)
Backing up data 7s done
Backup complete ∙ ffb2f619-1cb7-4a11-b3e2-7300aa513c6a
Bytes Uploaded: 0 B | Items Uploaded: 0 | Items Skipped: 0 | Errors: 0
Completed Backups:
ID ID Started At Duration Status Resource Owner
ffb2f619-1cb7-4a11-b3e2-7300aa513c6a 2023-10-20T10:23:35Z 40.096203016s Completed Teams Testing (meain)
```
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
This commit include a few things that improves the progresbars and other info messages are printed out during the CLI run.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
2nd last PR in the chain. Wires up the new sliding limiter into exchange backup create flows. I'll add some e2e regression tests to wrap it up.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3893
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3893
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3893
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3893
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Add logic to convert from msg(backup available in corso) to eml
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3893
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Currently we specify the service to rate limiter type bindings in `ProduceBackupCollections`. This means we lose the binding during `ConsumeBackupCollections` and default to using the exchange token bucket limiter for all services. Onedrive & SP are an exception here since we bind again during `streamItems`.
This PR moves the binding call one level above, to `do` scope so that the same bindings stay applied for the entirety of the backup op.
Note: This change has a slight side effect on integration tests which directly test `ProduceBackupCollections`. We'll default to using exchange token bucket limiter for such integ tests for all services. This shouldn't impact test runs too much since exch limts(16 per sec) or upto 1160 per min are still pretty good compared to 1200 per min for onedrive. Also, since these tests are short running & won't be doing a load test, we should be fine here.
If there is an objection, I can add rate limiter bindings manually for these `ProduceBackupCollections` tests.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
This continues the push towards having service-level handlers that know
how to perform different operations. It adds the helper functions that
will be used during restore operations to the existing handler code
This logic is not currently used nor does this PR change the restore
call path
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4254
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
introduces the delta tree in drive collections
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Issue(s)
* #4689
#### Test Plan
- [x] ⚡ Unit test
this is an alternative possible response from graph api when the client is updating the token in-flight. We normally catch this for item downloads, but are defaulting on permission downloads.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🐛 Bugfix
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Switch the folder name granularity to microseconds instead of seconds to avoid collisions with other concurrent CI test runs.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Move the token bucket limiter (`rate.Limiter`) package behind a corso owned `Limiter` interface. This allows callers to request tokens without knowing what type of limiter they are waiting on.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
replace all cases of new|wrap|stack(...).WithClues(ctx) with the flattened new|wrap|stackWC(ctx, ...) functions introduced in the latest clues bump.
Other changes added:
* remove WithClues builders when the error producer already called it.
* corrected some usages of ictx within loops.
no logic changes, just cleanup.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Allow callers of GetAddedRemovedItems to set the number of added items
they'd like returned from the call. Will eventually be leveraged for
Exchange preview backups
Doesn't change existing limits for pagers since everything uses the
default of zero for "get everything"
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
Add a killswitch for sliding window limiter. This is only relevant for exchange backups Setting this flag switches exchange backups to use the default token bucket rate limiter.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Adds `WaitN` support in sliding window limiter. WaitN blocks the request until all N tokens are acquired. It addresses a concern raised [here](https://github.com/alcionai/corso/pull/4636#discussion_r1389686910) in an earlier `WaitN` implementation by ensuring that all N tokens get served to each request atomically.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
This PR adds `Reset` support to limiter. In summary, it provides 2 things:
1. It resets limiter state & refills tokens to original capacity.
2. Reset is transparent to request senders, they can continue to call `Wait` even when reset is in progress. This is done using a new request level lock. It'll be useful towards `WaitN` implementation as well in a following PR.
Reason for adding `Reset`:
1. Corso CLI supports backing up the entire tenant. Since exchange service limits are defined on a per app per mailbox basis, if we don't reset the sliding window limiter in between n exchange resources, it'll artificially slow down n-1 resource backups.
2. This is because sliding window limiter front allows front loading the requests. This is not a big problem with token bucket limiter, since it refills tokens at a constant rate of 16 per second.
3. Ideally the rate limiter instance should be scoped to an exchange mailbox, and no `Reset` should be required. However, this would need some major refactoring of graph controller. Currently we use global rate limiter instances, and also the graph controller stays common for all resource backup operations. `Reset` allows us to keep using the global approach.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Bumps [github.com/microsoftgraph/msgraph-sdk-go](https://github.com/microsoftgraph/msgraph-sdk-go) from 1.24.0 to 1.25.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/microsoftgraph/msgraph-sdk-go/blob/main/CHANGELOG.md">github.com/microsoftgraph/msgraph-sdk-go's changelog</a>.</em></p>
<blockquote>
<h2>[1.25.0]- 2023-11-10</h2>
<ul>
<li>Weekly generation.</li>
</ul>
<h3>Changed</h3>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="b8836a2f64"><code>b8836a2</code></a> Generated models and request builders (<a href="https://redirect.github.com/microsoftgraph/msgraph-sdk-go/issues/615">#615</a>)</li>
<li><a href="c301b8df9d"><code>c301b8d</code></a> Merge pull request <a href="https://redirect.github.com/microsoftgraph/msgraph-sdk-go/issues/614">#614</a> from microsoftgraph/dependabot/go_modules/github.com/...</li>
<li><a href="2795816d4d"><code>2795816</code></a> Bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.8.0 to 1.9.0</li>
<li><a href="46ca257d24"><code>46ca257</code></a> Merge pull request <a href="https://redirect.github.com/microsoftgraph/msgraph-sdk-go/issues/610">#610</a> from microsoftgraph/dependabot/go_modules/github.com/...</li>
<li><a href="89106cfe7a"><code>89106cf</code></a> Bump github.com/microsoft/kiota-abstractions-go from 1.3.1 to 1.4.0</li>
<li><a href="9f7263506f"><code>9f72635</code></a> Merge pull request <a href="https://redirect.github.com/microsoftgraph/msgraph-sdk-go/issues/607">#607</a> from microsoftgraph/dependabot/go_modules/github.com/...</li>
<li><a href="eb710f57dc"><code>eb710f5</code></a> Bump github.com/microsoft/kiota-abstractions-go from 1.3.0 to 1.3.1</li>
<li>See full diff in <a href="https://github.com/microsoftgraph/msgraph-sdk-go/compare/v1.24.0...v1.25.0">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Wrapper for existing graph.ContainerResolver structs that returns items
in priority order in the Items() function. Also has logic to filter out
containers if desired
This can help with things like preview backups where we only want to
backup a subset of items because it allows us to ensure we'll see the
containers we care about most first
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
Concurrency limiter is stable and has been in use since several months. We had added a killswitch to disable it for safety reasons. It's not being used by SDK consumers, and hidden in corso cli.
This PR removes the `--disable-concurrency-limiter` flag & associated control option.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Bumps [github.com/aws/aws-xray-sdk-go](https://github.com/aws/aws-xray-sdk-go) from 1.8.2 to 1.8.3.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-xray-sdk-go/releases">github.com/aws/aws-xray-sdk-go's releases</a>.</em></p>
<blockquote>
<h2>v1.8.3</h2>
<p>Please refer <a href="https://github.com/aws/aws-xray-sdk-go/blob/master/CHANGELOG.md">change-log</a> for more details</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-xray-sdk-go/blob/master/CHANGELOG.md">github.com/aws/aws-xray-sdk-go's changelog</a>.</em></p>
<blockquote>
<h1>Release v1.8.3 (2023-11-13)</h1>
<h3>SDK Enhancements</h3>
<h3>SDK Bugs</h3>
<ul>
<li>Update AWS SDK for Go depdencies <a href="https://redirect.github.com/aws/aws-xray-sdk-go/pull/430">#PR 430</a></li>
<li>Fix HTTP2 rapid reset vulnerability <a href="https://redirect.github.com/aws/aws-xray-sdk-go/pull/428">#PR 428</a></li>
<li>Omit URL's password when stringifying URL for segment name <a href="https://redirect.github.com/aws/aws-xray-sdk-go/pull/422">#PR 422</a></li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="e9627da6f4"><code>e9627da</code></a> Merge pull request <a href="https://redirect.github.com/aws/aws-xray-sdk-go/issues/432">#432</a> from wangzlei/master</li>
<li><a href="e09dc9485b"><code>e09dc94</code></a> Merge branch 'aws:master' into master</li>
<li><a href="8d58e76a42"><code>8d58e76</code></a> fix replacement path</li>
<li><a href="4c99e077a6"><code>4c99e07</code></a> sync depdencies between sample app and sdk</li>
<li><a href="073fef2274"><code>073fef2</code></a> sync depdencies between sample app and sdk</li>
<li><a href="10e351fe4d"><code>10e351f</code></a> Merge pull request <a href="https://redirect.github.com/aws/aws-xray-sdk-go/issues/431">#431</a> from wangzlei/master</li>
<li><a href="6b717c755a"><code>6b717c7</code></a> Update changelog for 1.8.3</li>
<li><a href="5e4aa69293"><code>5e4aa69</code></a> Update changelog for 1.8.3</li>
<li><a href="d8c763246e"><code>d8c7632</code></a> Merge pull request <a href="https://redirect.github.com/aws/aws-xray-sdk-go/issues/430">#430</a> from wangzlei/master</li>
<li><a href="31f613777a"><code>31f6137</code></a> clean go.sum by go mod tidy</li>
<li>Additional commits viewable in <a href="https://github.com/aws/aws-xray-sdk-go/compare/v1.8.2...v1.8.3">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Adds a progress log in the kopia upload that logs every 10000 files that have been uploaded.
This is useful to track progress of large backups.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #4670
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Some code (e.x. Exchange) continues to use the results from a pager even
in the event some errors occur. These tests help ensure we have
consistent return values when we see an error during enumeration
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Rename these handlers to be base<Service>Handler so that we get some segmentation of functionality. This will allow us a bit of compile-time safety when it comes to accessing graph because the base handler doesn't contain a client instance. Essentially it allows us local access to things for operations like restore.
Future additions to the handler that require a client should wrap this base handler to provide that functionality. Export functions shouldn't be updated to be part of the new handler wrapper but should stay as part of the base handler so they continue to not have access to graph.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4254
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Follow up on #4557 [comment](https://github.com/alcionai/corso/pull/4557#pullrequestreview-1701971318) to consolidate test code for ease of use
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Bumps [golang.org/x/tools](https://github.com/golang/tools) from 0.14.0 to 0.15.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/golang/tools/releases">golang.org/x/tools's releases</a>.</em></p>
<blockquote>
<h2>gopls/v0.14.1</h2>
<p>This release contains just two changes:</p>
<ul>
<li>A workaround for a regression affecting some users of <code>GOPACKAGESDRIVER</code>: <a href="https://redirect.github.com/golang/go/issues/63751">golang/go#63751</a>, for example those using gopls with an older version of <a href="https://bazel.build/">Bazel</a>. When the <a href="https://pkg.go.dev/golang.org/x/tools/go/packages"><code>go/packages</code></a> driver is missing compiler or architecture information, gopls now assumes a default value rather than failing to load package information.</li>
<li>A fix for a minor bug in the <a href="https://github.com/golang/tools/releases/tag/gopls%2Fv0.14.0">new</a> "remove unused parameter" refactoring: <a href="https://redirect.github.com/golang/go/issues/63755">golang/go#63755</a>. Notably, this bug was discovered via an automated report from someone who had opted in to <a href="https://telemetry.go.dev/privacy">Go telemetry</a>.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="729e159c03"><code>729e159</code></a> go.mod: update golang.org/x dependencies</li>
<li><a href="38ed81a6b5"><code>38ed81a</code></a> gopls/internal/regtest/marker: porting extract tests</li>
<li><a href="bbf8380961"><code>bbf8380</code></a> gopls/internal/regtest/marker: use golden diffs for suggested fixes</li>
<li><a href="51df92b224"><code>51df92b</code></a> go/ssa: two minor cleanups</li>
<li><a href="e7fb31ad45"><code>e7fb31a</code></a> internal/cmd/deadcode: rename -format to -f</li>
<li><a href="c538b4e994"><code>c538b4e</code></a> internal/cmd/deadcode: add -whylive=function flag</li>
<li><a href="b753e58b84"><code>b753e58</code></a> internal/lsp/helper: fix misspelled "Code generated" comment</li>
<li><a href="2638d66336"><code>2638d66</code></a> internal/cmd/deadcode: omit package/func keywords in default output</li>
<li><a href="118c362a56"><code>118c362</code></a> gopls/internal/lsp/source: fix signatureHelp with pointer receivers</li>
<li><a href="4124316da0"><code>4124316</code></a> gopls/internal/lsp/cache: remove baseCtx from the View</li>
<li>Additional commits viewable in <a href="https://github.com/golang/tools/compare/v0.14.0...v0.15.0">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.16.0 to 0.17.0.
<details>
<summary>Commits</summary>
<ul>
<li><a href="b225e7ca6d"><code>b225e7c</code></a> http2: limit maximum handler goroutines to MaxConcurrentStreams</li>
<li>See full diff in <a href="https://github.com/golang/net/compare/v0.16.0...v0.17.0">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
> **Note**
> Automatic rebases have been disabled on this pull request as it has been open for over 30 days.
