828 Commits

Author SHA1 Message Date
Hitesh Pattanayak
1c18131122
replaces deleteList with errors fault bus (#4933)
replaces list deletion, on creation of list items failure, with fault bus. Thereby restores list with list items that could be restored.

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-26 12:29:40 +00:00
ashmrtn
87e41b20e0
JSON sanitizer and tests (#4925)
We've found that it's possible to get malformed JSON back from
the graph server and graph SDK. This results in errors while trying
to deserialize objects that are malformed JSON

This adds a simple JSON sanitizer and tests to make sure that even if
graph and the graph SDK happen to generate malformed JSON we can get
back to a state where we can deserialize it again. The tests also
print info when the received content differs from the original

This PR does not change any of the logic that corso uses during
backups or restores, it just adds sanitization code and tests

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-23 05:24:00 +00:00
Abhishek Pandey
ddf29e98dd
Add feature flag to disable lazy item reader (#4907)
<!-- PR description-->
AddDisableLazyItemReader disables lazy item reader, such that we fall
back to prefetch reader. This flag is currently only meant for groups
conversations backup. Although it can be utilized for other services
in future.

This flag should only be used if lazy item reader is the default choice
and we want to fallback to prefetch reader.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4862

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-22 23:59:00 +00:00
Hitesh Pattanayak
a1590e0d2f
enables sharepoint to use lists restore handler for lists ops (#4923)
enables sharepoint to use lists restore handler for lists ops
Changes previously approved in: 
- https://github.com/alcionai/corso/pull/4854
- https://github.com/alcionai/corso/pull/4910

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-22 17:18:24 +00:00
Hitesh Pattanayak
98e8cac374
enable restore path creation for sharepoint lists (#4922)
Enable restore path creation for sharepoint lists
Changes previously approved in: https://github.com/alcionai/corso/pull/4855

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-22 16:42:53 +00:00
Hitesh Pattanayak
dc47001cba
enables sharepoint to use lists backup handler for lists ops (#4920)
enables sharepoint to use lists backup handler for lists ops
Changes previously approved in:
-  https://github.com/alcionai/corso/pull/4786
- https://github.com/alcionai/corso/pull/4787
- https://github.com/alcionai/corso/pull/4909

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-22 14:26:23 +00:00
Hitesh Pattanayak
1270e64637
adds list client apis and pagers (#4917)
adds list client apis and pagers to fetch, create and delete lists and list relations.
Changes previously approved in PRs:
- https://github.com/alcionai/corso/pull/4785
- https://github.com/alcionai/corso/pull/4815
- https://github.com/alcionai/corso/pull/4852

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-22 08:02:32 +00:00
Hitesh Pattanayak
50fd28aab2
adds ListInfo details to SharepointInfo details (#4916)
adds ListInfo details to SharepointInfo details
Changes previously approved in PR: https://github.com/alcionai/corso/pull/4851

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

#### Issue(s)
#4754 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-22 06:50:34 +00:00
Keepers
8c0ac505a2
better parsing of graph odata errs (#4900)
parse the odataerr to a local struct that has
a more complete view of the data.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4685

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-21 21:39:31 +00:00
ashmrtn
c65527e5c4
Log and ignore sharing disabled errors when doing permissions restores (#4901)
We've encountered an error where the user previously had sharing for
one or more external users on an item and then did a backup. They
subsequently disabled sharing for the SharePoint site in either the
tenant-wide config or for the site in particular. Once sharing was
disabled, they attempted a restore which subsequently failed while
trying to restore sharing for the external user

This PR logs and ignores errors about sharing being disabled when
doing a restore

This has been tested manually

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-21 19:28:55 +00:00
ashmrtn
e46119a818
Remove options form service handler initializer (#4895)
Now that optiosn are being broken up a bit more passing a struct of a single type to the service handler initializer doesn't make much sense. Just remove the parameter since it was unused anyway.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issues

- #4896

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-20 18:49:28 +00:00
Keepers
7e25fe30f4
fix up cli tabular display and locationRef (#4594)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4536

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-20 03:23:13 +00:00
Keepers
92b996a3de
deltaURL stub refactor (#4865)
clean up the handling of deltaURL by using its own func instead of the id() func.  Also minimizes the delta stub constructor by using the suffix pattern used by id and folder constructors.

no logic changes, just a quick refactor.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-19 20:12:42 +00:00
Abin Simon
b896405e92
Add tests to check for invalid email addresses in eml export (#4881)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-19 19:41:42 +00:00
ashmrtn
fd056119dd
Add struct definitions and CLI helpers for backup options (#4860)
Define structs that will be used for backup options and add some CLI
helpers/tests to populate those structs with flag values

Rate limiter config is pulled out as a separate struct because it will
likely be used for backup and restore operations and it has values that
are [passed separately](505c06441a/src/internal/m365/backup.go (L232)) to the rate limiter config code

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-18 23:11:52 +00:00
Keepers
c58cd9302f
move /cli/config to /pkg (#4858)
Part 1 of two changes I've been carting around since the hackathon.
This PR moves the /cli/config package into /pkg/config.  This allows
sdk consumers to access corso configuration handling, intead of
requiring the CLI package to access it.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests


#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-18 18:24:33 +00:00
Abin Simon
83d9492b4f
Do not try to export items with error (#4867)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-18 07:08:57 +00:00
Keepers
44078e1db2
fix up test fullPath generation (#4833)
fixes up the generation of paths in drive testing so that paths are created correctly and consistently, even for values that will have escaped slashes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-12-15 21:44:26 +00:00
ashmrtn
4b64faef70
Add retries for posting item attachments (#4840)
Add exponential backoff and some retries when posting item attachments.

Manual testing was showing upwards of 50 retries in some cases when
~250-300 attachments total were being added. Of the retries, none
were at the max amount. However, all tests were for relatively small
datasets

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-13 23:59:42 +00:00
Vaibhav Kamra
d16528be50
Skip emails that cannot be retrieved (#4834)
Adds a skip condition for emails that can be enumerated but are not returned from the server (Exchange) because
it believes they are corrupt/invalid.

This handles the `ErrorInvalidRecipients` condition we believe is hit when exchange finds an email that pre-dates
M365 mailbox creation (either a server corruption or triggered by on-prem->M365 migration)

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-13 03:21:45 +00:00
Keepers
b94e5a677d
add test.t and pfx to test tree generators (#4777)
this is needed to standardize the presence of a path prefix in all test-helper trees, so that we can use standard test factory helpers for producing complete post-process data.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-12 17:44:23 +00:00
Keepers
9e36267303
minor mail attachment cleanup (#4816)
started down the path of changing behavior under error conditions involving server-side nil reference errors.  Issue turned out to be transient.  I'm commiting the cleanup without the error case handling.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-11 23:58:51 +00:00
jules
66ff1efc47
Adding new call to services middleware (#4823)
As the tittle suggests. Adding this call to the middleware used by clients.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E - added CI test
2023-12-08 20:28:02 +00:00
Keepers
8af0e31e4e
remove 'errors' from count keys (#4813)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
2023-12-08 19:45:39 +00:00
Keepers
54ba241fbe
require rootID on tree construction (#4746)
Turns out the root ID name isn't an appropriate match for establishing the root node.  Instead, the backup hander is now extended with a getRootFolder method and will pass the expected root folder ID into the tree's constructor func to ensure we establish the correct root node.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-07 23:54:45 +00:00
Vaibhav Kamra
29b37e9c15
Add a GetTeamByID API to lookup a team by ID (#4811)
Allows the caller to get Team information if they know a group is a team.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-07 03:39:26 +00:00
Keepers
27ff840be8
match specific error message for resourcre locked (#4806)
Matches a specific error message returned by graph when an account does not have support for microsoft teams.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-06 01:26:08 +00:00
Abhishek Pandey
3f98aa33de
Switch to custom drive items for backup operations (#4784)
<!-- PR description-->

Switch to using `custom.DriveItem` instead of `models.DriveItemable` during backups. There is a slight impact to restore as well, since backup and restore both use a few common interfaces e.g. `AugmentItemInfo`. 

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-12-05 22:48:02 +00:00
Abhishek Pandey
b5c9199695
Add custom drive item (#4783)
<!-- PR description-->

Add a `custom.DriveItem` struct which partially replicates `DriveItemable` and only stores the fields corso makes use of during a backup operation. 

Sorry about the PR size, I don't have a way to split this into multiple PRs. Also it should be trivial to review because it's simply copying over fields to another struct.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-05 21:40:11 +00:00
ashmrtn
02e9e1310e
Wire up setting min epoch duration (#4799)
Add the required wiring for SDK consumers to set the min epoch duration
(and eventually other persistent config info). This is just boiler-plate
with some tests at the various layers to make sure that layer is run as
expected. More nuanced tests about setting various config values are
left to the code in `internal/kopia/conn_test.go`

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4782

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-05 16:44:20 +00:00
Abhishek Pandey
3ee1a8d166
Minor url cache logging & counting improvements (#4801)
<!-- PR description-->

1. Move cache miss logs to debug, as it can be chatty sometimes. We will leverage the end of operation `URLCacheMiss` counter to get this information instead.
2. Counter renaming. 
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-05 02:18:13 +00:00
ashmrtn
7ef5a33edf
Allow setting the minimum epoch duration in kopia (#4798)
This PR lays the groundwork for changing persistent config info in
kopia. Specifically, it allows setting the minimum epoch duration (but
does so in a generic manner that can be expanded on later)

This PR only adds the code to actually do the configuration, it doesn't
add wiring to make this new function available to other callers

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4782

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-05 00:08:35 +00:00
Abhishek Pandey
2cd73e2d91
Remove sharepointids from delta query $select params (#4748)
<!-- PR description-->

We are not making use of `GetSharepointIds()` anymore. It was removed a month back, see [PR](https://github.com/alcionai/corso/pull/4436/files#diff-fc955b9b2e400cdda711ab2b6bf459845369691c5abc3973bbfaa1c83a664688L43). So we don't need to fetch this per item-property from graph during delta queries.

This will reduce overall mem usage since we don't need to store references to the sharepoint id map. It can also make delta queries go slightly faster since we are transferring & processing fewer bytes over network.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-12-04 18:07:35 +00:00
Keepers
6e572fd133
add and delete file support in the tree (#4724)
now that folder handling is complete, we can ingest items in the delta tree as well.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-12-04 16:37:53 +00:00
ashmrtn
08dfe6df66
Minio custom query params (#4763)
Add and wire up custom query parameters for minio to add extra fields
to S3 server access log output

** This switches to a custom minio version and updates kopia **

Diffs for the package updates (based on current versions used in corso):
* [minio](https://github.com/minio/minio-go/compare/v7.0.64...alcionai:minio-go:pass-custom-get-query-params-local?expand=1)
* [kopia](https://github.com/alcionai/kopia/compare/corsostaging-dir-iterator...custom-query-params)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-02 00:22:33 +00:00
Vaibhav Kamra
5a77db656e
Fix panic in GroupByID call (#4778)
`resp` is nil if there's an error and `getGroupFromResponse` will panic.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-01 19:33:19 +00:00
Keepers
37dcb952fb
add insufficient auth to graph, errs (#4776)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
2023-12-01 19:03:21 +00:00
Keepers
6307d8cbf6
prep groups handler for addition of conversations (#4583)
Makes modifications to the groups handlers to facilitate the incoming introduction of conversations backup handling. Changes include:
* new handler methods (canMakeDeltas and LocationRef)
* binding handlers to generics
* some naming normalization

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4536

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-01 00:40:37 +00:00
Keepers
d9c42f790c
introduce flat deletions to delta tree (#4719)
there are certain enumeration cases where tombstoning isn't the correct behavior, and we want to delete a folder from the tree entirely.  This primarily occurrs when we have a create->delete pair of markers either within or across enumerations.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🌻 Feature

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-12-01 00:12:57 +00:00
Abhishek Pandey
e1f3ffd0b3
Add configurable retries for 503 and 504 errors (#4750)
<!-- PR description-->

We use kiota retry handlers for 503 and 504 retries. We have a request from SDK users to configure number of retry attempts for 5xx errors. We already have configurability around retry attempts, plumbing it through to kiota retry handler.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-30 02:25:59 +00:00
Abhishek Pandey
5184fad18c
Minor retry handler test refactor (#4752)
<!-- PR description-->

Switch to configurable retries/timeouts/backoffs. Set up PR for https://github.com/alcionai/corso/pull/4750/files testing. 

With configurable retries, we can cut down the `TestRetryMiddleware_RetryResponse_maintainBodyAfter503` runtime from ~40seconds to around 1-2 seconds in the following PR #4750 



---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-30 01:21:08 +00:00
Abin Simon
9efe413e35
Sanity tests for email exports (#4741)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/4652

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-29 08:24:16 +00:00
Abin Simon
925c70d9d2
Fix sanity tests for SharePoint and Groups (#4740)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-11-29 07:56:09 +00:00
Keepers
6aff258c8b
add boilerplate for drive tree processing (#4716)
begins implementation of the drive delta tree support by adding boilerplate funcs for getting the backup data, and adding a framework of unit tests that will be used bring testing parity up to par with current tests.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-11-28 21:07:49 +00:00
Keepers
5a6ddde363
allow sdk users to set graph client timeouts (#4744)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-28 18:16:04 +00:00
ashmrtn
d36636285a
Use service-handler for restore operations (#4714)
Switch restore code path to use service-level handlers. Does a few
things:
* switches existing service-level functions to be methods on
  service-level handlers
* update interfaces as necessary
* moves some logic from old controller-level restore function to either
  the new handlers or the operation-level function
* removes old code

May be easiest to review by commit

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4254

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-11-28 16:47:45 +00:00
Abhishek Pandey
b4b24edca9
Add regression tests for drive pagers (#4620)
<!-- PR description-->


Adds regression tests for query params & headers used for delta/non delta requests.  Closes https://github.com/alcionai/corso/issues/4614


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4614

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-27 23:14:13 +00:00
ashmrtn
bcf290d628
Preview backup for drive-backed services (#4699)
Add logic and tests for preview backups in drive-backed services.
Does slightly change a few of the options for preview backup limits

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-11-18 00:46:36 +00:00
Keepers
d45842fe5c
add toggle for using the delta tree via cli (#4694)
#### Does this PR need a docs update or release note?

- [x]  No

#### Issue(s)

* #4690

#### Test Plan

- [x]  Unit test
2023-11-18 00:00:58 +00:00
Keepers
f4ed4d7250
bump xsync to v3 (#4704)
three changes
1. bumps the xsync package to v3
2. creates a common package for synced maps
3. replaces all xsync MapOf imports with the new common/syncd package.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-17 18:57:29 +00:00