477 Commits

Author SHA1 Message Date
ashmrtn
b8b1299514
Cleanup immutable ID config passing (#5013)
This value was present in the control.Options in the struct and
passed as a parameter. This PR removes the parameter so it's clear
the control.Options one is used

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #5012

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-17 00:51:11 +00:00
Keepers
6e50d5216e
remove builder.toServicePath from paths (#5028)
path.Build produces the same result with more standard and centralized behavior.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4025

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-16 16:42:49 +00:00
Keepers
dab53502ae
minor logging and cli output tweaks (#5016)
just touchups for supportability and tracking, plus a ux tweak here and there.  No real logic changes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-12 22:25:07 +00:00
Keepers
279a6dab3c
normalize observe usage, ensure deferred closers (#5006)
some housecleaning before reviewing deadlock issues

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
2024-01-11 22:42:10 +00:00
Keepers
b1af13b7b6
centralize core err interpretation (#4972)
now that we have a core error library, we can start swapping out the percolation of graph api errors with core error sentinels.  This PR begins that process by adding a centralized transformer to the graph service handlers that ensures all calls produce the same set of core errors according to some standard categorizations. This is just an initial step, and does not transfer all graph errors into core sentinels.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4685 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-10 17:25:25 +00:00
Hitesh Pattanayak
d113fa3926
store lists previouspath (#4938)
saves previous path for lists in storage. 

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 
#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2024-01-04 13:49:33 +00:00
Keepers
971d874462
introduce errs/core (#4897)
first step towards having a centralized set of error sentinels that can be passed around corso instead of re-using low level error sentinels like those found in the graph/errors.go file.

This PR works as a standalone, and only handles the lowest hanging fruit. SDK consumers will need to change their error enum references, but all api behavior should remain the same.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4685

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-03 18:13:41 +00:00
Abin Simon
b47fa8b14f
Ensure that we propagate errors if available (#4964)
<!-- PR description-->

Fixed all cases where we missed propagating errors. Also added a lint rule in CI to catch it.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-01-03 15:38:46 +00:00
Abhishek Pandey
2be9b3f301
Remove duplicated integration tests for groups (#4929)
<!-- PR description-->

* We had a few duplicated tests. Removing them.
* Adding conversations selector for group v9 bump tests. Although technically it's not needed, since we didn't have conversations support prior to v9.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-12-27 20:11:09 +00:00
Abhishek Pandey
c6b9615c7d
Integrate lazy item reader into conversations backup (#4921)
<!-- PR description-->




---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Closes https://github.com/alcionai/corso/issues/4862

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] 💚 E2E
2023-12-27 07:09:12 +00:00
ashmrtn
f63a6e9b4f
Some testing QoL functions (#4927)
Functions help check things like
  * are two items equal when taking into account nil, empty slices, and
    other zero-values?
  * is something fully populated?

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-26 18:52:50 +00:00
Abhishek Pandey
ddf29e98dd
Add feature flag to disable lazy item reader (#4907)
<!-- PR description-->
AddDisableLazyItemReader disables lazy item reader, such that we fall
back to prefetch reader. This flag is currently only meant for groups
conversations backup. Although it can be utilized for other services
in future.

This flag should only be used if lazy item reader is the default choice
and we want to fallback to prefetch reader.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4862

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-22 23:59:00 +00:00
Hitesh Pattanayak
98e8cac374
enable restore path creation for sharepoint lists (#4922)
Enable restore path creation for sharepoint lists
Changes previously approved in: https://github.com/alcionai/corso/pull/4855

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-22 16:42:53 +00:00
Hitesh Pattanayak
dc47001cba
enables sharepoint to use lists backup handler for lists ops (#4920)
enables sharepoint to use lists backup handler for lists ops
Changes previously approved in:
-  https://github.com/alcionai/corso/pull/4786
- https://github.com/alcionai/corso/pull/4787
- https://github.com/alcionai/corso/pull/4909

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-22 14:26:23 +00:00
Keepers
753ed1a075
use tree version in nightly tests (#4885)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x] 💚 E2E
2023-12-21 18:44:09 +00:00
ashmrtn
e46119a818
Remove options form service handler initializer (#4895)
Now that optiosn are being broken up a bit more passing a struct of a single type to the service handler initializer doesn't make much sense. Just remove the parameter since it was unused anyway.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issues

- #4896

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-20 18:49:28 +00:00
Abin Simon
b896405e92
Add tests to check for invalid email addresses in eml export (#4881)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-19 19:41:42 +00:00
Keepers
c58cd9302f
move /cli/config to /pkg (#4858)
Part 1 of two changes I've been carting around since the hackathon.
This PR moves the /cli/config package into /pkg/config.  This allows
sdk consumers to access corso configuration handling, intead of
requiring the CLI package to access it.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests


#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-18 18:24:33 +00:00
ashmrtn
505c06441a
Add config passing test that future PRs will leverage (#4857)
We're currently passing some backup options during repo connect which
complicates how SDK users can connect to a corso repo. This PR starts
to lay the groundwork for passing a discrete set of config parameters
to backup operations

This PR adds a test around config values passed into the
NewBackupOperation function. It checks that the config values stored
in the backup operation match what's passed in. It also makes sure
the passed in config parameters have all non-zero values so that
future changes don't silently pass the test when they shouldn't

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-14 22:41:00 +00:00
ashmrtn
3866ed2368
Exit test early if setup fails (#4842)
Reduce what needs searched through for debugging by exiting early.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup


#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-14 01:36:15 +00:00
ashmrtn
02e9e1310e
Wire up setting min epoch duration (#4799)
Add the required wiring for SDK consumers to set the min epoch duration
(and eventually other persistent config info). This is just boiler-plate
with some tests at the various layers to make sure that layer is run as
expected. More nuanced tests about setting various config values are
left to the code in `internal/kopia/conn_test.go`

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4782

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-05 16:44:20 +00:00
Keepers
6307d8cbf6
prep groups handler for addition of conversations (#4583)
Makes modifications to the groups handlers to facilitate the incoming introduction of conversations backup handling. Changes include:
* new handler methods (canMakeDeltas and LocationRef)
* binding handlers to generics
* some naming normalization

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4536

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-01 00:40:37 +00:00
ashmrtn
d36636285a
Use service-handler for restore operations (#4714)
Switch restore code path to use service-level handlers. Does a few
things:
* switches existing service-level functions to be methods on
  service-level handlers
* update interfaces as necessary
* moves some logic from old controller-level restore function to either
  the new handlers or the operation-level function
* removes old code

May be easiest to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4254

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-28 16:47:45 +00:00
ashmrtn
c48329f844
Return stats directly from restore function (#4713)
Rework restore return status so that later PRs will have a smaller diff

Instead of returning a status and then waiting on the message just return the restore stats directly. The status getter was only setup to wait for one item anyway and was setup to wait after the entire restore operation already completed (at end of
m365/restore.go:ConsumeRestoreCollections())

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4254

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-27 17:09:37 +00:00
Abhishek Pandey
d4519159e7
Move rate limiter reset to defer block (#4722)
Move `ResetLimiter()` call to defer block to ensure that it gets called even if the backup fails. Without this change, any following mailbox backups may get artificially throttled by limiter.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-22 17:02:02 +00:00
ashmrtn
8cf2c1b294
Cleaner way to disable merge bases during preview (#4711)
Switch to a cleaner way of disabling merge bases when making a preview backup. This sets the flag to not find merge bases instead of finding them and then dropping them.

Manually tested running a backup with `--preview` (local dev flag) and
`--disable-incrementals` and verified in the log that merge bases were
dropped 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-17 22:05:56 +00:00
Keepers
f4ed4d7250
bump xsync to v3 (#4704)
three changes
1. bumps the xsync package to v3
2. creates a common package for synced maps
3. replaces all xsync MapOf imports with the new common/syncd package.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-17 18:57:29 +00:00
ashmrtn
957a33b6d9
Finish wiring up Exchange preview backups (#4660)
This completes the wiring for Exchange preview backups. A few notable
points:
* Adds new config substruct(?) for limits on items/data
* Adds "reasonable defaults" (very small) for Exchange data categories
* Adds "important" and "skip" containers for Exchange data categories

All the above can be tweaked as we determine what values work best

Manually tested
1. regular backup
2. preview backup
3. regular backup

Verified that no merge base is used for the preview backup and the
merge base made in (1) is used for the backup in (3)

This feature is not exposed via CLI

May be easiest to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issues

Merge after:
* #4657
* #4607

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-17 01:46:17 +00:00
Keepers
9ecefd2569
use counter in exchange backup enum (#4661)
Increase proliferation of the count bus to record runtime stats.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-16 23:22:00 +00:00
Abin Simon
b46f242bc4
Make the cli colorful and informational (#4525)
This commit include a few things that improves the progresbars and other info messages are printed out during the CLI run.

![2023-10-20-17-17-09](https://github.com/alcionai/corso/assets/14259816/30a25832-24cb-48ef-944d-6aaced1cd62f)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-16 12:11:37 +00:00
Abhishek Pandey
f2102e55f6
Integrate sliding window limiter into exchange backup operation (#4680)
<!-- PR description-->

2nd last PR in the chain. Wires up the new sliding limiter into exchange backup create flows. I'll add some e2e regression tests to wrap it up.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-16 09:58:00 +00:00
Abhishek Pandey
6bc80ca3be
Move rate limiter ctx binding to backup operation scope (#4679)
Currently we specify the service to rate limiter type bindings in `ProduceBackupCollections`. This means we lose the binding during `ConsumeBackupCollections` and default to using the exchange token bucket limiter for all services. Onedrive & SP are an exception here since we bind again during `streamItems`. 

This PR moves the binding call one level above, to `do` scope so that the same bindings stay applied for the entirety of the backup op.

Note: This change has a slight side effect on integration tests which directly test `ProduceBackupCollections`. We'll default to using exchange token bucket limiter for such integ tests for all services. This shouldn't impact test runs too much since exch limts(16 per sec) or upto 1160 per min are still pretty good compared to 1200 per min for onedrive. Also, since these tests are short running & won't be doing a load test, we should be fine here.

If there is an objection, I can add rate limiter bindings manually for these `ProduceBackupCollections` tests. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-16 03:49:48 +00:00
Keepers
6fe6c9586d
retry item downloads on invalid token error (#4671)
this is an alternative possible response from graph api when the client is updating the token in-flight.  We normally catch this for item downloads, but are defaulting on permission downloads.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-16 01:46:34 +00:00
Keepers
4c72e9eab7
use clues new|wrap|stackWC() (#4684)
replace all cases of new|wrap|stack(...).WithClues(ctx) with the flattened new|wrap|stackWC(ctx, ...) functions introduced in the latest clues bump.

Other changes added:
* remove WithClues builders when the error producer already called it.
* corrected some usages of ictx within loops.

no logic changes, just cleanup.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-15 23:01:08 +00:00
ashmrtn
15418d0dda
Consolidate groups version bump tests (#4572)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4569

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-14 00:32:03 +00:00
Keepers
2ccd7da6a8
add a resource getter setter to newServiceHandler (#4666)
This sets us up to remove the service declaration
from the two-step repo creation + init/config, and allows sdk consumers to return to ignoring service until acting on the operation.
---

replacement branch for: https://github.com/alcionai/corso/pull/4494
2023-11-13 17:49:53 +00:00
Hitesh Pattanayak
40c7476e24
generate repository config name based on provider specific hash (#4639)
Enable user/client to be able to perform multiple backups in a concurrent manner irrespective of storage provider (s3/filesystem) for different tenants (anything that differentiates one account from another).

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4443 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-11 09:27:06 +00:00
ashmrtn
73112f695b
Move groups version bump tests to nightly (#4659)
Make version bump regression tests nightly tests instead of running on every push. Just code lift'n'shift with no logic changes.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-11 02:55:05 +00:00
jules
bb86610488
Getlicenseinfo (#4655)
Adds a call to get user total count of licenses. Also corrects one test ID which i hope doesn't break E2E tests (although it should if it does, ha)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-10 23:13:47 +00:00
Keepers
5362137116
expose graph options to sdk clients (#4653)
1. moves the m365/graph package from internal to pkg/services/api so that options are accessible to sdk users.
2. exposes graph.Options in the api client.Service call.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature
2023-11-10 19:26:59 +00:00
Keepers
1f756ce34f
support debug reading out delta files (#4629)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x] 💪 Manual
2023-11-10 16:19:16 +00:00
ashmrtn
16782d90d6
Don't allow merge bases for preview backups (#4628)
Preview backups may use different delta token query parameters
and we don't want them to get only changes from the last possible
backup. Avoid fine-grained tracking of this sort of stuff by just
discarding all merge bases

Keeping the bases as assist bases is fine because they'll only source
unchanged data when possible

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 00:54:07 +00:00
Keepers
49bbdfc096
add getPostIDs in conversations pager (#4578)
adds the getPostIDs func to ensure conversations
complies with standard data paging patterns

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4536

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-07 21:57:55 +00:00
Abhishek Pandey
ea6a57b2a5
Share count bus between graph clients and backup/restore operations (#4605)
<!-- PR description-->

With #4497 changes, any counters that we log in graph middlewares ( e.g. `APICallTokensConsumed` or `ThrottledAPICalls`) will no longer show up in `BackupOperation.Results.Counts`. 

This is because graph adapters and backup operation will be initialized with different count busses. Since the backup result counts are captured from the operation bus, we will lose out on any graph metrics captured in middleware layers. This PR unifies the busses. 

Ill also add a regression check for graph metrics shortly

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-07 06:41:11 +00:00
Keepers
3261eefda2
boilerplate introduction of the conversations api (#4570)
adds basic get requests for the conversations api set to the api package.  Plust some other minor change for formatting, correctness, or necessity.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4536 

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-06 20:05:52 +00:00
ashmrtn
e46cf645e5
Create and tag preview backups (#4595)
Add an option to request a preview backup and tag the resulting backup
as a preview if the flag is set. Preview backups must complete
successfully with no errors in order to be tagged

This does not update the item selection logic, so right now preview
backups will contain all items that normal backups do. Item selection
will be refined in upcoming PRs

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-11-01 17:22:49 +00:00
ashmrtn
74cf0ab737
Create service specific handlers that know how to run an export (#4491)
First step in reducing the number of places we have to check the service type manually. Create a way to get a handle to a service specific handler and implement exports for those handlers

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4254

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-01 15:55:30 +00:00
Keepers
5e0014307c
remove ctx embedding for counter (#4497)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-31 16:15:05 +00:00
ashmrtn
f61448d650
Groups version bump (#4561)
Bump the backup version and force a full backup if
there's a backup for teams/groups that has base(s)
from an older version of corso

This will avoid propagating older details formats
forward. Those formats don't have all the data
newer formats do

This is mostly a stop-gap, a more robust solution
can be added later

Manually tested that it forces a full backup

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4569

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-27 19:37:39 +00:00
ashmrtn
ba595a66c2
Remove Controller stats for export (#4500)
Export doesn't pull meaningful stats from the controller. Just remove
the collection/usage of these stats since it's confusing. Substitute the
check for not doing anything from a check on stats to a check on the
number of collections returned by kopia.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4254

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-10-26 21:09:17 +00:00