introduces the Chats service and Chats category
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #5061
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
Centralizing the suffixes so that group mailbox and future services can also use them.
* Move `.meta`, `.data`, `.dirmeta` definitions from `internal/m365/collection/drive/metadata` pkg to `pkg/services/m365/api/graph/metadata`.
* Adjust package names in misc places.
* No logic change.
* We should probably keep all metadata definitions and code in one place, i.e. relocate drive metadata definitions to `pkg/services/m365/api/graph/metadata`. This is a bigger change, we can do it later.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Adds the backup's data category to the cli output.
before:
```
ID Started At Duration Status Resource Owner
09e8d1e3-92d4-426d-9ca1-3cd063ce8bb9 2024-01-12T17:54:33Z 16.103569s Completed rfinders@10rqc2.onmicrosoft.com
893ef934-9403-4775-b667-09f77cd26e83 2024-01-12T17:55:22Z 5.66253s Completed rfinders@10rqc2.onmicrosoft.com
```
after:
```
ID Started at Duration Status Protected resource Data
09e8d1e3-92d4-426d-9ca1-3cd063ce8bb9 2024-01-12T17:54:33Z 16.103569s Completed rfinders@10rqc2.onmicrosoft.com Contacts,Emails,Events
893ef934-9403-4775-b667-09f77cd26e83 2024-01-12T17:55:22Z 5.66253s Completed rfinders@10rqc2.onmicrosoft.com Emails
```
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3748
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
Will be used in later PRs to help make long-running streamstore
snapshots safer to create if they ever go past the kopia checkpoint
interval
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
merge after:
* #5091
#### Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
- includes GetList in list restore handler
- includes GetListsByCollisionKey in list restore handler
- updates mock list restore handler
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
Posts are always sent to the group email address, along with additional
recipients if any. Currently, we don't have a way to get the unique
recipient list for individual posts due to graph api limitations.
Store the group mail address so that we can use it to populate the 'To'
field during Post -> EML exports.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
path.Build produces the same result with more standard and centralized behavior.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4025
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
just touchups for supportability and tracking, plus a ux tweak here and there. No real logic changes.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Skips backup of non-recoverable list templates.
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Enable restore path creation for sharepoint lists
Changes previously approved in: https://github.com/alcionai/corso/pull/4855
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
adds ListInfo details to SharepointInfo details
Changes previously approved in PR: https://github.com/alcionai/corso/pull/4851
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #4536
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Part 1 of two changes I've been carting around since the hackathon.
This PR moves the /cli/config package into /pkg/config. This allows
sdk consumers to access corso configuration handling, intead of
requiring the CLI package to access it.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Makes modifications to the groups handlers to facilitate the incoming introduction of conversations backup handling. Changes include:
* new handler methods (canMakeDeltas and LocationRef)
* binding handlers to generics
* some naming normalization
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #4536
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Previously we were printing a table for a single item which, for one
was unnecessary, but this also make the output of a backup more
compact.
Before:
```
Logging to file: /home/meain/.cache/corso/logs/2023-10-20T10-21-23Z.log
Connecting to M365 done
Connecting to repository 1s done
Backing Up ∙ Teams Testing (meain)
Discovering items to backup 27s done
Libraries (TeamsTestingmeain) done (found 0 items)
Libraries (TeamsTestingmeain-Shared0) done (found 0 items)
Libraries (TeamsTestingmeain-Private2) done (found 0 items)
Messages 7s done (found 6 channels)
Backing up data 7s done
Backup complete ∙ 7b40dd40-f808-4d57-8e39-b4553e48dc5d
ID Bytes Uploaded Items Uploaded Items Skipped Errors
7b40dd40-f808-4d57-8e39-b4553e48dc5d 0 B 0 0 0
Completed Backups:
ID Started At Duration Status Resource Owner
7b40dd40-f808-4d57-8e39-b4553e48dc5d 2023-10-20T10:21:32Z 36.632747912s Completed Teams Testing (meain)
```
After:
```
Connecting to M365 done
Connecting to repository 1s done
Backing Up ∙ Teams Testing (meain)
Discovering items to backup 31s done
Libraries (TeamsTestingmeain) done (found 0 items)
Libraries (TeamsTestingmeain-Shared0) done (found 0 items)
Libraries (TeamsTestingmeain-Private2) done (found 0 items)
Messages 9s done (found 6 channels)
Backing up data 7s done
Backup complete ∙ ffb2f619-1cb7-4a11-b3e2-7300aa513c6a
Bytes Uploaded: 0 B | Items Uploaded: 0 | Items Skipped: 0 | Errors: 0
Completed Backups:
ID ID Started At Duration Status Resource Owner
ffb2f619-1cb7-4a11-b3e2-7300aa513c6a 2023-10-20T10:23:35Z 40.096203016s Completed Teams Testing (meain)
```
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
adds basic get requests for the conversations api set to the api package. Plust some other minor change for formatting, correctness, or necessity.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #4536
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Allow the backup model to tell us what type it was tagged with. This
will make logic for base selection easier since it won't have to worry
about extracting the type anymore.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Adds two-step processing to the html previews for channel messages and replies. First, all inline attachments are replaced with the string `[attachment:name]`.
Second, remaining html is stripped out, leaving only plaintext.
This transformation is applied to both the exported content and the preview content in details.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #4546
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
builds out more details for channel messages and replies
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
#### Issue(s)
* #3988
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Move the Reasoner implementation from the kopia package to the identity
package. This will help avoid import cycles if we want to start
persisting Reason information in the backup model.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
some additional logging context for collection scope.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
This was an accidental mis-use of the skipped item pattern. Items deleted during in flight during backup due to race conditions do not count as permanent
skips in the way that other skipped items do.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#4044
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #3988
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Still a few pending items like CLI handling and restoring specific items. Will be handled in a follow up.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3992
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
updaing the path package to the current naming convention. No logic changes.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
enables nightly cli e2e tests, operations layer integration tests, and sanity tests for groups and teams.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #3989
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Not sure if we wanna merge it as it might generate way too many conflicts, but this should help us add a linter in CI. If we are good, I'll add something that can do lints for this in a follow up PR.
Super hacky, but this fix was created using `while true ; do tree-grepper -q go '(argument_list "," @nope .)' | tail -n1| awk -F: "{print \$1,\"+\"\$2\" -c ':norm \$xJZZ'\"}" | xargs vim ; done`.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3654
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
info.itemType already looks through each service entry to find the item type. A service nil check is redundant. Adds a qol func for asking if an itemInfo is a driveish item.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #3990
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
A bunch of places where we missed handling groups
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3989
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
Adds groups channel message filters to group selectors. Includes scopes for message creation time, last response time, and message created by.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3989
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Shuffle around some logic for details merging so that
we always attempt to extract a LocationRef from the
backup base entry that's currently being examined.
A LocationRef should always be available from either
the LocationRef field in the details entry (newer
backups) or by extracting it from the RepoRef (older
backups)
Manually tested incremental backups for exchange in
the following scenarios:
1. v0.3.0 backup (calendars use IDs in RepoRef) ->
incremental with this patch -> incremental with
this patch
1. v0.2.0 backup (exchange uses folder names in
RepoRef) -> incremental with this patch ->
incremental with this patch
The above tests should cover the cases where:
* base backup details don't have LocationRef for
exchange items
* base backup details have LocationRef for exchange
items
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#3716
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3989
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
CLI changes for groups.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3990
*
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
I've needed to catch gotchas that arise from contributors adding a value in the middle of an iota list, not to mention have dealt with prior bugs that happened the same way, now too many times to feel safe about its usage.
This PR removes the use of iota from all const declarations. The intent is to not allow the use of iota within the codebase.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3993
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Munging code originally existed so that SDK
users would get unique ShortRefs if an item
moved or was renamed. However, we no longer
need to support that so this removes that
munging code
Manually tested making a new backup with this
PR where the merge base was a backup made prior
to this PR. A folder with a subfolder and items
was moved between the two backups. Backup
details for the new backup contained all the
expected entries
Restore selector logic should be unaffected
as that assumes the user has passed in a
ShortRef that was previously printed to the
CLI by `corso backup details`. Input is
compared against the ShortRef stored in the
entry, `ShortRef()` is not called on any `Path`
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#4012
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
No longer required by SDK users and not exposed directly to CLI users unless they're looking at the JSON output from details. This field is not documented anywhere though so there's no guarantees that we technically need to uphold with it.
Manually tested:
* restoring from a backup that had this field
* making a fresh backup without this field when the merge base did have this field
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#3986
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Only code movement, no code changes.
Moved services to `/internal/m365/services/{onedrive,sharepoint,exchange}`
Moved collections to `/internal/m365/collection/{drive,site}`
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
This PR adds changes to persist backup models for assist backups. Primary reason for doing this is to ensure we don't lose corso extension data(deets) for items which were downloaded and processed by kopia during this backup op.
Note: This is 1/n PRs, and will not be merged into main until all n are reviewed, given we are touching state here.
**Changes in this PR:**
1. Error handling changes to ensure assist backups get persisted
2. Introduce tags for assist/merge backups. These tags will be used for filtering during base selection process.
3. Don't persist backups if they are not assist backup, and they have non zero errors.
4. Remove `LabelForceNoBackupCreation` criteria for not persisting a backup. More details are in [PR comment](https://github.com/alcionai/corso/pull/3905/files#r1278030140).
5. Switch kopia failure from non-recoverable to recoverable. More details are in [PR comment](https://github.com/alcionai/corso/pull/3905/files#r1278019992).
**What are assist backups?**
Below conditions must be true for a backup operation to qualify as an assist backup.
1. new deets were produced during this backup operation
2. we have a valid snapshot ID
3. we don't have any non-recoverable errors
4. we have recoverable errors
**A merge backup satisfies the following conditions:**
1. Is not an assist backup
2. No primary or recoverable errors occurred during the backup operation.
Tests will be added in later PRs.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* internal
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Moves the kopia.Reasoner interface out of the
kopia package and into pkg/backup/identity.
No logical changes here, just renaming and movement.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3993
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Adding in some basic boilerplate for groups service.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
Collection of fixes to get tests passing again including:
* ensure expected sets are tracked for the lifetime of the struct
* add check that will fail if there's no expected deets to compare against
* minor fixups to path building when reading in an entire details file
* trim down data sent for patch operations to only the data that's changed
* update how expected items are gathered -- just pull it from graph directly
* update expected written items for OneDrive -- most likely due to matching both test folders (e.x. `corso_incr_test_1` and `corso_incr_test_1/corso_incr_test_1`)
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#3777
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
* Integrates corso extensions into onedrive/sharepoint library item backup flows.
* Also includes integration tests for OD/SP, unit tests for collections.
* Includes a small fix for `rw *backupStreamReader Close()`
Remaining things which will be covered in later PRs:
* extension tests with incremental backups
* Observability related changes for extensions
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* internal
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
a variety of small updates that came
from manual testing of restore with various
collision and destination combinations.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #3562
#### Test Plan
- [x] 💪 Manual
<!-- PR description-->
Introduces interfaces for corso extensions, which is basically wrappers around `io.ReadCloser`. Will add integration in follow up PRs.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
renames /internal/connector to /internal/m365. No logic changes in this PR. Only the dir rename, import renames, and one linter shadowing rename.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1996
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Related to https://github.com/alcionai/corso/pull/3140 . We were editing the deets in place and returning a pointer. This is fine as long as we were only calling it once, but https://github.com/alcionai/corso/pull/3480 created an additional call for this which causes the `konwnFolders` to be appended twice.
Old backups will still have incorrect data, but an incremental backup on top of it should fix up the data.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Fixes https://github.com/alcionai/corso/issues/3555
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
Currently the backup output show,
- total backup size - size of files + size of metafiles
- no of files - no of files + no of metafile
With this change the output of backup command shows size of only files and count include only file count.
NOTE: all current three services results will be impacted here.
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3304
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
