1453 Commits

Author SHA1 Message Date
Abhishek Pandey
7a7802e271 Add custom drive item 2023-12-01 02:10:29 -08:00
Abhishek Pandey
05dee79560 clone strings to lose references to in mem stores 2023-11-29 21:49:18 -08:00
Abhishek Pandey
178ae0d465 Remove remaining refs to msft stores 2023-11-28 22:24:48 -08:00
Abhishek Pandey
b5a400b896 Remove references to CreatedBy map 2023-11-28 20:34:19 -08:00
Abhishek Pandey
993b648eff Put back existing code 2023-11-20 21:05:07 -08:00
Abhishek Pandey
27383e950e Flatten everything 2023-11-20 20:23:09 -08:00
Abhishek Pandey
b444ed328e flatten file and folder 2023-11-18 01:14:48 -08:00
Abhishek Pandey
d5ac19275f fix 2023-11-18 00:45:08 -08:00
Abhishek Pandey
09a66ffae8 Add folder 2023-11-18 00:42:45 -08:00
Abhishek Pandey
f1f1be5b9c Add custom drive itemable 2023-11-18 00:37:39 -08:00
Abhishek Pandey
b1cf23bee3 remove sizeof pkg 2023-11-17 20:30:38 -08:00
Abhishek Pandey
ba8e5828b8 4min pause before gc 2023-11-17 19:59:03 -08:00
Abhishek Pandey
b148be72dc 1GB gomemlim 2023-11-17 19:19:16 -08:00
Abhishek Pandey
aa2522d03c Only do 1 iteration 2023-11-17 17:51:19 -08:00
Abhishek Pandey
52b020c416 profile 2023-11-17 15:57:43 -08:00
ashmrtn
8e978cce37
Improve logging/variable grouping for exchange preview backup (#4705)
Just shuffling some variables and log statements around. No logic or
test changes beyond that

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-17 03:48:53 +00:00
Abhishek Pandey
0fcbf75f6f
Add connection timeout retries to item retry handler (#4706)
We were missing a retry for connection timeout issues. Adding it.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-17 02:13:16 +00:00
ashmrtn
957a33b6d9
Finish wiring up Exchange preview backups (#4660)
This completes the wiring for Exchange preview backups. A few notable
points:
* Adds new config substruct(?) for limits on items/data
* Adds "reasonable defaults" (very small) for Exchange data categories
* Adds "important" and "skip" containers for Exchange data categories

All the above can be tweaked as we determine what values work best

Manually tested
1. regular backup
2. preview backup
3. regular backup

Verified that no merge base is used for the preview backup and the
merge base made in (1) is used for the backup in (3)

This feature is not exposed via CLI

May be easiest to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issues

Merge after:
* #4657
* #4607

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-17 01:46:17 +00:00
Keepers
2a85b61213
clean up unused graph error sentinels (#4683)
A handful of the error sentinel values in graph/errors.go were only around to comply with an old expectation that IsErr funcs have a matching sentinel.  That assumption is out of date, and there's no point in keeping around unused sentinels.

This PR makes three changes to clean up that issue:
1. remove unused sentinels (unused is qualifed as "not produced").
2. fix external pacakges which referenced those errors in tests alone by replacing those sentinels with odata errors.
3. centralize odata error test code to help support upstream package odata error creation.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-16 23:59:38 +00:00
Keepers
9ecefd2569
use counter in exchange backup enum (#4661)
Increase proliferation of the count bus to record runtime stats.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-16 23:22:00 +00:00
ashmrtn
7a21424ca7
Minor cleanup for drive backup code (#4698)
Just a few minor updates to stats logic to reset the counters if the delta is reset and also only count items if they weren't previously seen and counted.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-16 20:14:31 +00:00
ashmrtn
1cbabdb73a
Exchange restore service level handler functions (#4693)
Add service-level restore handlers for exchange

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4254

merge after:
* #4687
* #4651

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-16 18:22:33 +00:00
Abin Simon
b46f242bc4
Make the cli colorful and informational (#4525)
This commit include a few things that improves the progresbars and other info messages are printed out during the CLI run.

![2023-10-20-17-17-09](https://github.com/alcionai/corso/assets/14259816/30a25832-24cb-48ef-944d-6aaced1cd62f)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-16 12:11:37 +00:00
Abhishek Pandey
f2102e55f6
Integrate sliding window limiter into exchange backup operation (#4680)
<!-- PR description-->

2nd last PR in the chain. Wires up the new sliding limiter into exchange backup create flows. I'll add some e2e regression tests to wrap it up.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-16 09:58:00 +00:00
Abin Simon
dfea154cfe
Imrpove message -> eml conversion (#4651)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-16 07:39:34 +00:00
Abin Simon
c3b7246ee9
Improvements and tests for eml conversion (#4644)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3893

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-16 07:03:36 +00:00
Abin Simon
140402361a
Add support for email exports (#4642)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3893

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-16 05:43:05 +00:00
Abin Simon
2dfe22dd0c
Converter for msg to eml (#4640)
Add logic to convert from msg(backup available in corso) to eml

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3893

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-16 04:23:04 +00:00
Abhishek Pandey
6bc80ca3be
Move rate limiter ctx binding to backup operation scope (#4679)
Currently we specify the service to rate limiter type bindings in `ProduceBackupCollections`. This means we lose the binding during `ConsumeBackupCollections` and default to using the exchange token bucket limiter for all services. Onedrive & SP are an exception here since we bind again during `streamItems`. 

This PR moves the binding call one level above, to `do` scope so that the same bindings stay applied for the entirety of the backup op.

Note: This change has a slight side effect on integration tests which directly test `ProduceBackupCollections`. We'll default to using exchange token bucket limiter for such integ tests for all services. This shouldn't impact test runs too much since exch limts(16 per sec) or upto 1160 per min are still pretty good compared to 1200 per min for onedrive. Also, since these tests are short running & won't be doing a load test, we should be fine here.

If there is an objection, I can add rate limiter bindings manually for these `ProduceBackupCollections` tests. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-16 03:49:48 +00:00
ashmrtn
51f44c2988
Add functions used for restore to existing service-level handlers (#4687)
This continues the push towards having service-level handlers that know
how to perform different operations. It adds the helper functions that
will be used during restore operations to the existing handler code

This logic is not currently used nor does this PR change the restore
call path

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4254

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-16 02:48:54 +00:00
Keepers
dbdd3f236c
add delta tree (#4692)
introduces the delta tree in drive collections

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-11-16 02:16:00 +00:00
Keepers
6fe6c9586d
retry item downloads on invalid token error (#4671)
this is an alternative possible response from graph api when the client is updating the token in-flight.  We normally catch this for item downloads, but are defaulting on permission downloads.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-16 01:46:34 +00:00
ashmrtn
cb4a8cb5c7
Use microsecond granularity for exchange tests (#4695)
Switch the folder name granularity to microseconds instead of seconds to avoid collisions with other concurrent CI test runs.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-16 01:06:59 +00:00
Abhishek Pandey
ff5d83c723
Move various limiter types behind a common interface (#4678)
Move the token bucket limiter (`rate.Limiter`) package behind a corso owned `Limiter` interface. This allows callers to request tokens without knowing what type of limiter they are waiting on.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-16 00:13:34 +00:00
Keepers
4c72e9eab7
use clues new|wrap|stackWC() (#4684)
replace all cases of new|wrap|stack(...).WithClues(ctx) with the flattened new|wrap|stackWC(ctx, ...) functions introduced in the latest clues bump.

Other changes added:
* remove WithClues builders when the error producer already called it.
* corrected some usages of ictx within loops.

no logic changes, just cleanup.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-15 23:01:08 +00:00
ashmrtn
ea2bf19bd1
Wire up config for pager result limit (#4657)
Allow callers of GetAddedRemovedItems to set the number of added items
they'd like returned from the call. Will eventually be leveraged for
Exchange preview backups

Doesn't change existing limits for pagers since everything uses the
default of zero for "get everything"

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-15 22:33:07 +00:00
Abhishek Pandey
bc9d258ca0
Add WaitN support to limiter (#4675)
Adds `WaitN` support in sliding window limiter. WaitN blocks the request until all N tokens are acquired.  It addresses a concern raised [here](https://github.com/alcionai/corso/pull/4636#discussion_r1389686910) in an earlier `WaitN` implementation by ensuring that all N tokens get served to each request atomically. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-15 17:55:17 +00:00
Abhishek Pandey
2e6cb20a59
Add limiter reset functionality (#4674)
This PR adds `Reset` support to limiter. In summary, it provides 2 things:
1. It resets limiter state & refills tokens to original capacity.
2. Reset is transparent to request senders, they can continue to call `Wait` even when reset is in progress. This is done using a new request level lock. It'll be useful towards `WaitN` implementation as well in a following PR. 

Reason for adding `Reset`:
1. Corso CLI supports backing up the entire tenant. Since exchange service limits are defined on a per app per mailbox basis, if we don't reset the sliding window limiter in between  n exchange resources, it'll artificially slow down n-1 resource backups. 
2. This is because sliding window limiter front allows front loading the requests. This is not a big problem with token bucket limiter, since it refills tokens at a constant rate of 16 per second. 
3. Ideally the rate limiter instance should be scoped to an exchange mailbox, and no `Reset` should be required. However, this would need some major refactoring of graph controller. Currently we use global rate limiter instances, and also the graph controller stays common for all resource backup operations. `Reset` allows us to keep using the global approach. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-14 22:18:34 +00:00
ashmrtn
e85e33c58e
Create a ContainerResolver that returns items in priority order (#4607)
Wrapper for existing graph.ContainerResolver structs that returns items
in priority order in the Items() function. Also has logic to filter out
containers if desired

This can help with things like preview backups where we only want to
backup a subset of items because it allows us to ensure we'll see the
containers we care about most first

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-14 20:42:36 +00:00
Abhishek Pandey
02108fab95
Remove unused control option for disabling concurrency limiter (#4677)
<!-- PR description-->

Concurrency limiter is stable and has been in use since several months. We had added a killswitch to disable it for safety reasons. It's not being used by SDK consumers, and hidden in corso cli. 

This PR removes the `--disable-concurrency-limiter` flag & associated control option.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-14 19:45:34 +00:00
Vaibhav Kamra
477fc43485
Add progress logging to uploader (#4672)
Adds a progress log in the kopia upload that logs every 10000 files that have been uploaded.

This is useful to track progress of large backups.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #4670 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-14 08:18:27 +00:00
ashmrtn
15418d0dda
Consolidate groups version bump tests (#4572)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4569

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-14 00:32:03 +00:00
ashmrtn
bb27b8e44d
Rename service handlers (#4571)
Rename these handlers to be base<Service>Handler so that we get some segmentation of functionality. This will allow us a bit of compile-time safety when it comes to accessing graph because the base handler doesn't contain a client instance. Essentially it allows us local access to things for operations like restore.

Future additions to the handler that require a client should wrap this base handler to provide that functionality. Export functions shouldn't be updated to be part of the new handler wrapper but should stay as part of the base handler so they continue to not have access to graph.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4254

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-13 23:08:14 +00:00
Keepers
2ccd7da6a8
add a resource getter setter to newServiceHandler (#4666)
This sets us up to remove the service declaration
from the two-step repo creation + init/config, and allows sdk consumers to return to ignoring service until acting on the operation.
---

replacement branch for: https://github.com/alcionai/corso/pull/4494
2023-11-13 17:49:53 +00:00
Hitesh Pattanayak
40c7476e24
generate repository config name based on provider specific hash (#4639)
Enable user/client to be able to perform multiple backups in a concurrent manner irrespective of storage provider (s3/filesystem) for different tenants (anything that differentiates one account from another).

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4443 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-11 09:27:06 +00:00
ashmrtn
73112f695b
Move groups version bump tests to nightly (#4659)
Make version bump regression tests nightly tests instead of running on every push. Just code lift'n'shift with no logic changes.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-11 02:55:05 +00:00
jules
bb86610488
Getlicenseinfo (#4655)
Adds a call to get user total count of licenses. Also corrects one test ID which i hope doesn't break E2E tests (although it should if it does, ha)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-10 23:13:47 +00:00
Keepers
80feeace3b
ensure invalid deltas result in new state collections (#4623)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-10 21:45:27 +00:00
ashmrtn
c85d364bcd
Allow limiting or cancelling pager enumeration (#4625)
Add the ability to cancel a pager partway through enumeration. This
will allow us to later request a specific number of items from the
pager. In those cases we should ideally fetch one page beyond what's
required to get the requested number of items (due to the pager
running in another goroutine)

Does not change the result set or add the ability to request a certain
number of items right now

May be easier to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 20:33:27 +00:00
Keepers
5362137116
expose graph options to sdk clients (#4653)
1. moves the m365/graph package from internal to pkg/services/api so that options are accessible to sdk users.
2. exposes graph.Options in the api client.Service call.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature
2023-11-10 19:26:59 +00:00