111 Commits

Author SHA1 Message Date
Abin Simon
444e26fd6d Use /content instead of download link from delta 2024-01-31 22:43:05 +05:30
Abhishek Pandey
8133da3087
Centralize metadata suffix definitions (#5134)
<!-- PR description-->

Centralizing the suffixes so that group mailbox and future services can also use them.

* Move `.meta`, `.data`, `.dirmeta` definitions from `internal/m365/collection/drive/metadata` pkg to `pkg/services/m365/api/graph/metadata`.
* Adjust package names in misc places.
* No logic change. 
* We should probably keep all metadata definitions and code in one place, i.e. relocate drive metadata definitions to `pkg/services/m365/api/graph/metadata`. This is a bigger change, we can do it later.
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-01-26 22:06:04 +00:00
Abin Simon
214d8a6030
Fixing handling of permissions for external users (#5097)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/5046

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-24 19:32:03 +00:00
Keepers
d501a30761
use the tree-based drive backup by standard (#4980)
reverses the drive backup process toggle by making the tree-based backup standard, and the old, linear enumeration the toggleable fallback.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4690

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-23 18:57:11 +00:00
Keepers
6e50d5216e
remove builder.toServicePath from paths (#5028)
path.Build produces the same result with more standard and centralized behavior.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4025

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-16 16:42:49 +00:00
Keepers
1694581e26
make a not found core error (#4975)
make a canonical 'not found' error for our core sentinels. This replaces the graph.DeletedInFlight error, which acted as an interpretation of "not found".

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4685

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2024-01-12 01:29:39 +00:00
Keepers
279a6dab3c
normalize observe usage, ensure deferred closers (#5006)
some housecleaning before reviewing deadlock issues

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
2024-01-11 22:42:10 +00:00
Abin Simon
ca50b1e908
Filter out non existent entities in permissions and link shares (#4955)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-08 13:44:25 +00:00
Abin Simon
a52b085cea
Update handling of permissions restore failures (#4954)
- All permissions failures are now recoverable to the granularity of a single permission or link share
- Failure to delete child permissions because the permission was not restored on parent will now not produce any warnings

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2024-01-08 10:32:41 +00:00
Abin Simon
b47fa8b14f
Ensure that we propagate errors if available (#4964)
<!-- PR description-->

Fixed all cases where we missed propagating errors. Also added a lint rule in CI to catch it.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2024-01-03 15:38:46 +00:00
Hitesh Pattanayak
50fd28aab2
adds ListInfo details to SharepointInfo details (#4916)
adds ListInfo details to SharepointInfo details
Changes previously approved in PR: https://github.com/alcionai/corso/pull/4851

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-22 06:50:34 +00:00
Abin Simon
c3b15a00bd
Note about anonymous link shares (#4905)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-22 06:13:28 +00:00
ashmrtn
c65527e5c4
Log and ignore sharing disabled errors when doing permissions restores (#4901)
We've encountered an error where the user previously had sharing for
one or more external users on an item and then did a backup. They
subsequently disabled sharing for the SharePoint site in either the
tenant-wide config or for the site in particular. Once sharing was
disabled, they attempted a restore which subsequently failed while
trying to restore sharing for the external user

This PR logs and ignores errors about sharing being disabled when
doing a restore

This has been tested manually

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-21 19:28:55 +00:00
Keepers
753ed1a075
use tree version in nightly tests (#4885)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x] 💚 E2E
2023-12-21 18:44:09 +00:00
Keepers
92b996a3de
deltaURL stub refactor (#4865)
clean up the handling of deltaURL by using its own func instead of the id() func.  Also minimizes the delta stub constructor by using the suffix pattern used by id and folder constructors.

no logic changes, just a quick refactor.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-19 20:12:42 +00:00
Abin Simon
b896405e92
Add tests to check for invalid email addresses in eml export (#4881)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-19 19:41:42 +00:00
Keepers
c58cd9302f
move /cli/config to /pkg (#4858)
Part 1 of two changes I've been carting around since the hackathon.
This PR moves the /cli/config package into /pkg/config.  This allows
sdk consumers to access corso configuration handling, intead of
requiring the CLI package to access it.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests


#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-18 18:24:33 +00:00
Keepers
44078e1db2
fix up test fullPath generation (#4833)
fixes up the generation of paths in drive testing so that paths are created correctly and consistently, even for values that will have escaped slashes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-12-15 21:44:26 +00:00
Keepers
e3363aaa46
fix up failures found when manually testing (#4825)
correct the tree post-process traversal, ensuring the root folder doesn't appear twice in the path.  And provide the drive name to the collection so that the backup details can utilize it.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4689

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-12-14 23:28:16 +00:00
Keepers
dc3cfd1ec3
clean up drive unit test helpers (#4805)
Adds some cleanups to the drive unit test helpers.  Goals were as follows:
1. remove idx() and namex() funcs, replace with id() and name()
2. minimize factory duplication in helper_test.go
3. improve readability in id() and name() calls by adding file and folder variants to handle the 99% of cases we use them in.

No logic changes in this PR.  Only test func updates.
---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-12-14 17:02:44 +00:00
Keepers
b94e5a677d
add test.t and pfx to test tree generators (#4777)
this is needed to standardize the presence of a path prefix in all test-helper trees, so that we can use standard test factory helpers for producing complete post-process data.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-12 17:44:23 +00:00
Keepers
eb4f238502
add prev-path updates to tree (#4762)
introduces a func to the delta tree that allows a caller to set the previous-path on a folder by ID.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-08 18:14:06 +00:00
Keepers
54ba241fbe
require rootID on tree construction (#4746)
Turns out the root ID name isn't an appropriate match for establishing the root node.  Instead, the backup hander is now extended with a getRootFolder method and will pass the expected root folder ID into the tree's constructor func to ensure we establish the correct root node.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-07 23:54:45 +00:00
Abhishek Pandey
3838258a57
Remove graph sdk store references in url cache item map (#4800)
<!-- PR description-->

Use `Clone()` on item properties so that we don't hold on to graph in-memory store references.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-07 01:10:55 +00:00
Abhishek Pandey
3f98aa33de
Switch to custom drive items for backup operations (#4784)
<!-- PR description-->

Switch to using `custom.DriveItem` instead of `models.DriveItemable` during backups. There is a slight impact to restore as well, since backup and restore both use a few common interfaces e.g. `AugmentItemInfo`. 

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-05 22:48:02 +00:00
Keepers
047d46ea53
add new funcs and tidy up drive limiter (#4734)
adds some new functions to the drive limiter that will be used specifically in the tree-based backup process. Also updates the limiter tests to have separate versions for the tree and non-tree variations.  In this PR, the tree variation will definitely fail.  The next PR will be focused on backup process corrections needed to ensure that the limit handling is correct according to the existing tests.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-05 22:18:49 +00:00
Abhishek Pandey
3ee1a8d166
Minor url cache logging & counting improvements (#4801)
<!-- PR description-->

1. Move cache miss logs to debug, as it can be chatty sometimes. We will leverage the end of operation `URLCacheMiss` counter to get this information instead.
2. Counter renaming. 
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-05 02:18:13 +00:00
Keepers
6e572fd133
add and delete file support in the tree (#4724)
now that folder handling is complete, we can ingest items in the delta tree as well.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-12-04 16:37:53 +00:00
Abhishek Pandey
79a37bfb91
Remove duplicate recoverable err during item download (#4766)
<!-- PR description-->

We are logging a recoverable error in 2 places for the same failure during lazy item download. See the [code](6307d8cbf6/src/internal/kopia/upload.go (L228)) for the other location.  For e.g. below 2 recoverable errors are referring to the same graph 400 failure. 

`running backup: recoverable error 1 of 6: downloading item content: retrieving expired item: getting item: Invalid request: Invalid request`
`
running backup: recoverable error 2 of 6: kopia reported error: fetching item content: retrieving expired item: getting item: Invalid request: Invalid request`

Removing the duplicate since it can be confusing to the end user & devs.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-01 18:33:01 +00:00
Keepers
d9c42f790c
introduce flat deletions to delta tree (#4719)
there are certain enumeration cases where tombstoning isn't the correct behavior, and we want to delete a folder from the tree entirely.  This primarily occurrs when we have a create->delete pair of markers either within or across enumerations.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🌻 Feature

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-12-01 00:12:57 +00:00
Keepers
6aff258c8b
add boilerplate for drive tree processing (#4716)
begins implementation of the drive delta tree support by adding boilerplate funcs for getting the backup data, and adding a framework of unit tests that will be used bring testing parity up to par with current tests.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-11-28 21:07:49 +00:00
Keepers
bbf5350f6e
add basic folder operations to tree (#4696)
adds SetFolder and AddTombstone operations
to the drive delta tree.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-11-28 17:20:22 +00:00
ashmrtn
a777359abe
Add logging like in exchange backup path (#4710)
Log both the given limits and the effective limits so we can figure out if something went wrong.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-18 06:21:08 +00:00
ashmrtn
fd91d36c91
Standardize naming for preview default limits (#4709)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-18 03:43:34 +00:00
ashmrtn
a3fe9d8d2e
Test default drive limits are applied when expected (#4708)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-18 01:56:35 +00:00
ashmrtn
bcf290d628
Preview backup for drive-backed services (#4699)
Add logic and tests for preview backups in drive-backed services.
Does slightly change a few of the options for preview backup limits

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-18 00:46:36 +00:00
Keepers
f4ed4d7250
bump xsync to v3 (#4704)
three changes
1. bumps the xsync package to v3
2. creates a common package for synced maps
3. replaces all xsync MapOf imports with the new common/syncd package.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-17 18:57:29 +00:00
Keepers
2a85b61213
clean up unused graph error sentinels (#4683)
A handful of the error sentinel values in graph/errors.go were only around to comply with an old expectation that IsErr funcs have a matching sentinel.  That assumption is out of date, and there's no point in keeping around unused sentinels.

This PR makes three changes to clean up that issue:
1. remove unused sentinels (unused is qualifed as "not produced").
2. fix external pacakges which referenced those errors in tests alone by replacing those sentinels with odata errors.
3. centralize odata error test code to help support upstream package odata error creation.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-16 23:59:38 +00:00
Keepers
9ecefd2569
use counter in exchange backup enum (#4661)
Increase proliferation of the count bus to record runtime stats.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-16 23:22:00 +00:00
ashmrtn
7a21424ca7
Minor cleanup for drive backup code (#4698)
Just a few minor updates to stats logic to reset the counters if the delta is reset and also only count items if they weren't previously seen and counted.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-16 20:14:31 +00:00
Abin Simon
b46f242bc4
Make the cli colorful and informational (#4525)
This commit include a few things that improves the progresbars and other info messages are printed out during the CLI run.

![2023-10-20-17-17-09](https://github.com/alcionai/corso/assets/14259816/30a25832-24cb-48ef-944d-6aaced1cd62f)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-16 12:11:37 +00:00
Abhishek Pandey
6bc80ca3be
Move rate limiter ctx binding to backup operation scope (#4679)
Currently we specify the service to rate limiter type bindings in `ProduceBackupCollections`. This means we lose the binding during `ConsumeBackupCollections` and default to using the exchange token bucket limiter for all services. Onedrive & SP are an exception here since we bind again during `streamItems`. 

This PR moves the binding call one level above, to `do` scope so that the same bindings stay applied for the entirety of the backup op.

Note: This change has a slight side effect on integration tests which directly test `ProduceBackupCollections`. We'll default to using exchange token bucket limiter for such integ tests for all services. This shouldn't impact test runs too much since exch limts(16 per sec) or upto 1160 per min are still pretty good compared to 1200 per min for onedrive. Also, since these tests are short running & won't be doing a load test, we should be fine here.

If there is an objection, I can add rate limiter bindings manually for these `ProduceBackupCollections` tests. 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-16 03:49:48 +00:00
Keepers
dbdd3f236c
add delta tree (#4692)
introduces the delta tree in drive collections

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-11-16 02:16:00 +00:00
Keepers
6fe6c9586d
retry item downloads on invalid token error (#4671)
this is an alternative possible response from graph api when the client is updating the token in-flight.  We normally catch this for item downloads, but are defaulting on permission downloads.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix


#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-16 01:46:34 +00:00
Keepers
4c72e9eab7
use clues new|wrap|stackWC() (#4684)
replace all cases of new|wrap|stack(...).WithClues(ctx) with the flattened new|wrap|stackWC(ctx, ...) functions introduced in the latest clues bump.

Other changes added:
* remove WithClues builders when the error producer already called it.
* corrected some usages of ictx within loops.

no logic changes, just cleanup.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-15 23:01:08 +00:00
Keepers
80feeace3b
ensure invalid deltas result in new state collections (#4623)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-10 21:45:27 +00:00
ashmrtn
c85d364bcd
Allow limiting or cancelling pager enumeration (#4625)
Add the ability to cancel a pager partway through enumeration. This
will allow us to later request a specific number of items from the
pager. In those cases we should ideally fetch one page beyond what's
required to get the requested number of items (due to the pager
running in another goroutine)

Does not change the result set or add the ability to request a certain
number of items right now

May be easier to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-10 20:33:27 +00:00
Keepers
5362137116
expose graph options to sdk clients (#4653)
1. moves the m365/graph package from internal to pkg/services/api so that options are accessible to sdk users.
2. exposes graph.Options in the api client.Service call.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature
2023-11-10 19:26:59 +00:00
Keepers
85d13b382a
some minor logging touchups (#4617)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-10 18:40:57 +00:00
Keepers
1f756ce34f
support debug reading out delta files (#4629)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x] 💪 Manual
2023-11-10 16:19:16 +00:00