11 Commits

Author SHA1 Message Date
Keepers
f4847404c4
add per-container item collision enumeration (#3629)
populates the item collision cache generators for all exchange data types.  Next we'll use these to compare the items being restored for collision before making post requests, by populating this cache at the top of each collection restore process.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-22 20:54:06 +00:00
Abhishek Pandey
c8ae50cb2e
Refresh URL cache using previous delta token (#3649)
<!-- PR description-->

Currently we do full delta enumeration during cache refresh events. It has some disadvantages as listed in https://github.com/alcionai/corso/issues/3482 .

This PR allows us to instantiate the URL cache with a delta token. This delta token is identical to the one used during onedrive item enumeration.

The URL cache might do multiple delta queries(once every hour) using the same delta token.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)
* https://github.com/alcionai/corso/issues/3482

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-06-21 18:57:43 +00:00
Abhishek Pandey
479f114514
Integrate URL cache into onedrive backup flows (#3542)
Changes:

* Attach URL cache to a collection only if it's parent drive has < 300k items. Higher item count reduces the ROI of using cache, and can even degrade performance in severe cases. We will stick to per item GETs for those scenarios, until we figure out a longer term solution.
* 
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3069

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-20 23:18:01 +00:00
Keepers
416383a99c
cascade restoreCfg collision policy into onedrive (#3623)
Adds collision policy handling to onedrive item posts.  This allows us to override the default "replace" behavior that currently returns a 409 for the creation endpoint, in case we want to use skip (ie: fail) or copy handling.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-16 00:07:09 +00:00
Keepers
5184920b52
add foldermatcher to handler (#3565)
Missed adding the foldermatcher to the handlers
for onedrive and sharepoint during the api refactor.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-15 18:28:31 +00:00
Keepers
46eedc0ad1
add default restore config normalizer (#3613)
Adds a func that normalizes the restore config in case inputs don't match accepted or expected values.

Also moves DefaultTestRestoreConfig out of /tester and into /control/testdata to avoid circular dependency issues.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
2023-06-15 16:04:42 +00:00
Abhishek Pandey
dd19b484c8
Add more tests for url cache (#3593)
PR contents
1. Address a corner case where cache may be half filled ( e.g. scenario: delta query failing after a few pages). Empty the cache on any delta failures
2. Add unit tests 

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-06-15 07:48:03 +00:00
Keepers
bd0f6f9769
final cleanup for renaming/movement (#3612)
This last step in the cleanup has two goals: first to minimize the number of arbitrarily named and located files so that code is better condensed and co-located.  This is all just file renaming and a minor amount of code copy-pasting.

The second is to create a centralized package to own the ColInfo type structs that we use to both stub out test data, and also generate factory data for cmds.  The current ownership is haphazard, and while this movement is a little more condensed, it's still jumping through some weird hoops to get things to work.  Treat it as one good step forward, and we'll have to return to polish it another time.  At least it'll be separated from the m365 folder at large this way, and more easily identified as supporting design rather than production usage.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-14 16:50:17 +00:00
Keepers
466698f096
fix restore func misnaming in onedrive (#3616)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996
2023-06-14 16:09:44 +00:00
Keepers
ce72acbcc1
auto-log recoverable errors with stack (#3598)
automatically log when we add a recoverable error or a skipped item to fault.  This log will include a stack trace of the call from the location of the logged recoverable.  Clues does not have a method for pulling a stack trace out of an error yet; that can be added at a future date.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-13 22:18:18 +00:00
Keepers
2f6d731993
rename connector -> m365 (#3600)
renames /internal/connector to /internal/m365.  No logic changes in this PR.  Only the dir rename, import renames, and one linter shadowing rename.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-13 18:35:39 +00:00