Switch restore code path to use service-level handlers. Does a few
things:
* switches existing service-level functions to be methods on
service-level handlers
* update interfaces as necessary
* moves some logic from old controller-level restore function to either
the new handlers or the operation-level function
* removes old code
May be easiest to review by commit
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4254
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Rework restore return status so that later PRs will have a smaller diff
Instead of returning a status and then waiting on the message just return the restore stats directly. The status getter was only setup to wait for one item anyway and was setup to wait after the entire restore operation already completed (at end of
m365/restore.go:ConsumeRestoreCollections())
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4254
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
2nd last PR in the chain. Wires up the new sliding limiter into exchange backup create flows. I'll add some e2e regression tests to wrap it up.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
First step in reducing the number of places we have to check the service type manually. Create a way to get a handle to a service specific handler and implement exports for those handlers
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4254
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Update the BackupBases API to return BackupBase
structs that contain both the snapshot and backup
model instead of having separate functions for
each. Minor logic updates to accommodate these
changes
This PR also updates tests and mock code for the
new API
Suggest viewing by commit
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#3943
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Have a way to gather stats about the exported data.
Users can now call `ExportOperation.GetStats()` at the end of the run to get the stats for the operations. The data will be in the format `map[path.CategoryType]data.KindStats` whre `KindStats` is:
```go
type KindStats struct {
BytesRead int64
ResourceCount int64
}
```
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4311
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
hands the backup resource into the drive collection for the handler to use to record as the siteID
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #3988
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
the addition of the drive expand requires onedrive permissions in order to retrieve sites by ID. This causes some users to violate customer contract by requiring unwanted permissions. This change allows configuration of the getbyID call to optionally include the drive extend, which enables the backup process to return to the permission set as expected from v0.12
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #4337
#### Test Plan
- [ ] 💪 Manual
- [ ] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3990
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
various tidbits of data cleanup before moving forward with adding export behavior to groups.
* move duplicate collections mocks into data/mock
* move the export collection struct into pkg/export (to prevent future duplicates in the next PR)
* rename export.Collection to Collectioner, because it's an interface.
* some other non-logic rearrangement
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3991
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
1. Renamed `IsBackupRunnable` to `IsServiceEnabled` & extended to restore operation. Removed `checkServiceEnabled`. Reasons:
- The above 2 functions were doing the same thing. Removed `checkServiceEnabled` in favor of `IsBackupRunnable` since we want this check to be as soon as possible during backup/restore op initialization
- Renamed `IsBackupRunnable` to `IsServiceEnabled`, because a) we are only doing service enabled checks right now, b) common code that can be used for both restore & backup.
2. Wire corso code to use new helpers in internal/m365/common.go
3. Note: SDK wiring and related integ tests will be added in a follow up PR
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3844
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
A Stream is a continuous transmission of data.
An item is a single structure. Crossing the two
definitions generates confusion.
Primarily code movement/renaming. Though there
is also some reduction/replacement of structs
where we'd made a variety of testable Item implementations
instead of re-using the generic mock.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
1. Tests which try out all combinations of failurePolicy vs fault errors
2. Incremental tests for deets propagation from assist/merge backups
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
Adds another inject container-of-things to hold
common properties used by backup collection producers.
No logic changes, just code movement, renames, and placing things into structs.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
If the restore configuration specifies a protected resource as a restore target, use that as the destination for the restore. First step is to ensure the provided target can be retrieved and identified.
---
#### Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
Core logic for exporting data from OneDrive
Next: https://github.com/alcionai/corso/pull/3820
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3670
* https://github.com/alcionai/corso/pull/3797
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Adds api handlers for creating document libraries in sharepoint. This is the first step in allowing us to restore drives that were deleted between backup and restore.
---
#### Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Introduces a counting bus, and threads it into restore operations so that we can count the number of
collision skips that occur.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
This last step in the cleanup has two goals: first to minimize the number of arbitrarily named and located files so that code is better condensed and co-located. This is all just file renaming and a minor amount of code copy-pasting.
The second is to create a centralized package to own the ColInfo type structs that we use to both stub out test data, and also generate factory data for cmds. The current ownership is haphazard, and while this movement is a little more condensed, it's still jumping through some weird hoops to get things to work. Treat it as one good step forward, and we'll have to return to polish it another time. At least it'll be separated from the m365 folder at large this way, and more easily identified as supporting design rather than production usage.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#1996
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
renames /internal/connector to /internal/m365. No logic changes in this PR. Only the dir rename, import renames, and one linter shadowing rename.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1996
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
