Fixes tests failing due to change in PR #5060
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
- https://github.com/alcionai/corso/actions/runs/7736081427
- https://github.com/alcionai/corso/actions/runs/7751227007
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
* Add EML exports for group mailbox.
* Tested E2E manually along with unit tests added in this PR.
* Will follow it up with a sanity test PR.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
* Add the serialized `Postable` to EML converter code for group mailboxes and associated tests.
* We should converge this with `Messageable` code at some point with a small translator function which converts `Postable` to `Messageable`. Reason being that posts are a subset of messages. Although I feel it's a bit premature right now. Once we have tested this in prod a bit, and we know for a fact that posts are indeed a true subset of messages, we can do this convergence.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
seems like a lot of code, but this is 95% boilerplate additions copied from groups collections packages, with a find-replace for names.
Some noteworthy differences:
* teamsChats does not handle metadata, so all metadata, delta, and previous path handling was removed
* teamsChats does not produce tombstones
* chats are never deleted, so no "removed" items are tracked
* all chats gets stored at the prefix root, so no "containers" are iterated, and therefore only one collection is ever produced.
This means that, overall, while the boilerplate here is still the same, it's much reduced compared to similar packages.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #5062
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Unclear if this is precisely the path we want to take to handle possible
partial repo initialization so feel free to leave suggestions if a
different behavior would be better
Update the repo init logic to continue if the repo is marked as already existing. If it already exists then the process will try to update persistent config info in the repo. If the repo already exists but some connection credential is incorrect then stack errors such that both the reason for the connect failure and the fact that the repo already exists are both visible.
Manually tested attempting to initialize a repo that was already
initialized but using a different passphrase and got the output:
```text
Error: initializing repository: a repository was already initialized with that configuration: connecting to kopia repo: unable to create format manager: invalid repository password: repo already exists: repository already initialized
exit status 1
```
Attempting to initialize a repo again using the same passphrase resulted
in a success message with no errors reported
Unfortunately I can't really add unit tests to ensure the config ends up
the way we expect because I have no way to trigger an error partway through
kopia's init logic (or even just after it) without some non-trivial code
refactoring
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
* Each post now has a `.data` and `.meta` file.
* I've only made changes to groups lazy reader to keep this PR short. Note that channels don't have meta/data file concepts, we don't want to accidentally enable this for channels. Given channels doesn't use lazy reader, this is safe to do short term.
* I'll be adding small follow up PRs to 1) Make sure channels files don't get assigned `.data` suffix. 2) Support for data and meta files for prefetch conversations backup.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
adds documentation for sharepoint lists.
should go in after: https://github.com/alcionai/corso/pull/5048
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
makes backup of lists explicit and not by default.
**Backup whole site**:
```
corso backup create sharepoint --site https://10rqc2.sharepoint.com/sites/CorsoCI
Completed Backups:
ID Started at Duration Status Protected resource Data
4a324fd8-dcfc-4833-961f-316d11da34e9 2024-01-30T05:45:24Z 21.530965673s Completed https://10rqc2.sharepoint.com/sites/CorsoCI Libraries
corso backup details sharepoint --backup 4a324fd8-dcfc-4833-961f-316d11da34e9
ID ItemName Library ParentPath Size Owner Created Modified
15204b287c5c wat.jpg More Documents test 6.1 kB RFinders@10rqc2.onmicrosoft.com 2023-03-24T20:33:46Z 2023-03-24T20:33:46Z
69f63c25d3f1 test-file.txt Documents Corso_Test_od_restore_and_backup_multi_30-Jan-2024_05-06-04.380107/b 387 B 2024-01-30T05:06:09Z 2024-01-30T05:06:09Z
c2d55ecb218f test-file.txt Documents Corso_Test_od_restore_and_backup_multi_30-Jan-2024_05-06-04.380107/folder-a 131 B 2024-01-30T05:06:11Z 2024-01-30T05:06:12Z
5d38274759fd test-file.txt Documents Corso_Test_od_restore_and_backup_multi_30-Jan-2024_05-06-04.380107 87 B 2024-01-30T05:06:06Z 2024-01-30T05:06:07Z
b5c075c00737 test-file.txt Documents Corso_Test_od_restore_and_backup_multi_30-Jan-2024_05-06-04.380107/folder-a/b/folder-a 387 B 2024-01-30T05:06:16Z 2024-01-30T05:06:17Z
...
...
```
**Backup lists explicitely**:
```
corso backup create sharepoint --site https://10rqc2.sharepoint.com/sites/CorsoCI --data lists
Completed Backups:
ID Started at Duration Status Protected resource Data
cf480086-2094-4184-82be-26e2c5c64288 2024-01-30T05:50:35Z 23.605968852s Completed https://10rqc2.sharepoint.com/sites/CorsoCI Lists
corso backup details sharepoint --backup cf480086-2094-4184-82be-26e2c5c64288
ID List Items Created Modified
85d453fd0f13 Corso_Test_Sanity_2024-01-30_05-31-05_e6be57ae-4236-44aa-b506-a5801377f937 20 2024-01-30T05:31:15Z 2024-01-30T05:31:18Z
f8919f072c7c Corso_Test_Sanity_Restore_20240130_053009_Corso_Test_Sanity_2024-01-30_05-29-33_4fed1e50-f045-4ebb-8a61-8fb608ac9673 20 2024-01-30T05:30:26Z 2024-01-30T05:30:29Z
821df8ab207d Corso_Test_Sanity_Restore_20240130_053147_Corso_Test_Sanity_2024-01-30_05-29-33_b44ce328-106c-4880-8e37-e0bbb510ed4d 20 2024-01-30T05:31:58Z 2024-01-30T05:32:02Z
721f0aadf07f new-list-name 1 2024-01-19T17:44:26Z 2024-01-29T04:05:42Z
...
...
```
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
enables lists restore
should go in after https://github.com/alcionai/corso/pull/5121
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
handles single & multiple values for metadata columns
Similar to `Hyperlink` and `Column` columns, `Metadata` column too is unrecognizable from GRAPH API response. Hence identifying from the field column names.
`Metadata` fields are like tags. A `Metadata` fields can be configured to hold multiple values/tags
**Original List with `Metadata` column (Department) with single value/tag**:
**Restored List with `Metadata` column with single value/tag**:
**Original List with `Metadata` column (Department) with multiple value/tag**:
**Restored List with `Metadata` column with multiple value/tag**:
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
#5084#5108
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Disable sending requests to microsoft servers to get API tokens when
using the gock client. This is accomplished by passing in a mock that
always returns no error for auth requests.
**This PR does not go through and make existing tests using gock unit
tests instead of integration tests. That will need to be done
separately**
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #5124
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
introduces the Chats service and Chats category
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #5061
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
provisions to identify fields with multiple value.
Some fields can hold multiple values and single value based on a property called `allowMultipleValues`.
For example, `Lookup` column:
```
"lookup": {
"allowMultipleValues": true,
"allowUnlimitedLength": false,
"columnName": "Title",
"listId": "21b45bf2-e495-4582-b114-839577ff8e4f"
}
```
But `choice` columns, even though allows to set multiple choices/value, does not have that particular field `allowMultipleValues` to indicate.
So in this PR we are trying determine the same by the stored values while restoring.
**Original list with `choice` column in site**:
**Restored list with `choice` column in site**
The color does not come through though
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
#5108
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Instead of relying on strings pulled from the graph SDK to identify the types of the items we're dealing with use the type of the interface in golang.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #5124
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
Catches issues where we use `WC` variant of clues when `ctx` is passed and as well as cases where we don't use WC variant when `ctx` is not passed in.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Just make sure we can call Close() on the repository struct even if we have made a connection to kopia or initialized all the internal fields.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Add the wiring to actually call the new config verify command during
maintenance
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
merge after:
* #5117
* #5118
* #5122
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Bumps [github.com/arran4/golang-ical](https://github.com/arran4/golang-ical) from 0.2.3 to 0.2.4.
<details>
<summary>Commits</summary>
<ul>
<li><a href="51fa6f1213"><code>51fa6f1</code></a> Merge pull request <a href="https://redirect.github.com/arran4/golang-ical/issues/85">#85</a> from brackendawson/panic</li>
<li><a href="46e2a5c0ed"><code>46e2a5c</code></a> Exclude fuzz testing from pre-1.18 toolchains</li>
<li><a href="a8f0586c90"><code>a8f0586</code></a> fix panic when param value has incomplete escape sequence</li>
<li><a href="3631125a31"><code>3631125</code></a> fix panic when property ends without colon or value</li>
<li><a href="681cc6e62c"><code>681cc6e</code></a> fix panic for missing property param operator</li>
<li><a href="804f4d3436"><code>804f4d3</code></a> Merge pull request <a href="https://redirect.github.com/arran4/golang-ical/issues/84">#84</a> from arran4/pull-66-add-todo-journal</li>
<li><a href="cf8d1b371d"><code>cf8d1b3</code></a> Ignore explicitly</li>
<li><a href="1e96c15957"><code>1e96c15</code></a> Consistent receivers.</li>
<li><a href="9e30bdf5f0"><code>9e30bdf</code></a> Ignore explicitly</li>
<li><a href="55df13ec27"><code>55df13e</code></a> Consistent todo receiver</li>
<li>Additional commits viewable in <a href="https://github.com/arran4/golang-ical/compare/v0.2.3...v0.2.4">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Bumps [jarallax](https://github.com/nk-o/jarallax) from 2.1.4 to 2.2.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/nk-o/jarallax/blob/master/CHANGELOG.md">jarallax's changelog</a>.</em></p>
<blockquote>
<h2>[2.2.0] - Jan 27, 2024</h2>
<ul>
<li>updated video worker:
<ul>
<li>added support for private Vimeo videos hash in URL</li>
<li>fixed <code>play</code> method play when <code>endTime</code> reached</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="f8060f98d1"><code>f8060f9</code></a> v2.2.0</li>
<li><a href="2df482e478"><code>2df482e</code></a> updated video-worker</li>
<li><a href="afa456f38b"><code>afa456f</code></a> Update npm-publish.yml</li>
<li>See full diff in <a href="https://github.com/nk-o/jarallax/compare/v2.1.4...v2.2.0">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Switch to using alerts and the fault bus instead of errors. Hopefully
this will make it easier to ensure this verify code doesn't fail
maintenance overall and that callers can consume the info in a
standardized manner
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
merge after:
* #5117
* #5118
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Add function that returns errors if it finds issues with common config
info in the kopia repo. Parameters that are currently checked are:
* kopia global policy:
* kopia snapshot retention is disabled
* kopia compression matches the default compression for corso
* kopia scheduling is disabled
* object locking:
* maintenance and blob config blob parameters are consistent (i.e. all
enabled or all disabled)
Note that tests for this will fail until alcionai/clues#40 is
merged and clues is updated in corso
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
merge after:
* #5117
* alcionai/clues#40
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Latest clues patch fixes bug where the result of `clues.Stack(clues.Stack(nil))` was
not considered nil
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
hopefully fixes clobbering issues for end-of-run tables
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #5113
#### Test Plan
- [x] 💪 Manual
<!-- PR description-->
Centralizing the suffixes so that group mailbox and future services can also use them.
* Move `.meta`, `.data`, `.dirmeta` definitions from `internal/m365/collection/drive/metadata` pkg to `pkg/services/m365/api/graph/metadata`.
* Adjust package names in misc places.
* No logic change.
* We should probably keep all metadata definitions and code in one place, i.e. relocate drive metadata definitions to `pkg/services/m365/api/graph/metadata`. This is a bigger change, we can do it later.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Basic validation checks that make sure maintenance and retention options are consistent with each other. This at least makes sure retention is either enabled and maintenance extends locks or nothing is locked at all.
Going to be part of a set of PRs for validating the kopia config
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
metadata is too much data to be useful to an end user, meanwhile err is getting clobbered.
before
```
Error: Unable to complete backup
Message: <nil>
Metadata:{"assist_bases":"0","backup_id":"40940295-3611-42fb-ac27-5b80691ba793","base_backup_id":"832fff09-4988-4a3e-a0d0-4e51b197ea14","can_use_metadata":"true","can_use_previous_backup":"true","categories":"[Chats]","clues_trace":"c3bbd0e6","collection_count":"2","disable_assist_backup":"false","incremental":"true","merge_bases":"1","repo_ref":"fb8afbaa-e94c-4ea5-8a8a-24aff04d7874/chats/311d2a89-4c80-4da4-86ef-a08d8d6f0f94/chats/19:311d2a89-4c80-4da4-86ef-a08d8d6f0f94_817c2506-de4a-4795-971e-371ea75a03ed@unq.gbl.spaces","resource_owner_id":"311d2a89-4c80-4da4-86ef-a08d8d6f0f94","resource_owner_name":"rfinders@10rqc2.onmicrosoft.com","resource_owner_selected":"rfinders@10rqc2.onmicrosoft.com","service":"service(5)","tenant_id":"fb8afbaa-e94c-4ea5-8a8a-24aff04d7874"}
```
after
```
Error: Unable to complete backup
Cause: running backup: merging details: merging merge backup base details: getting updated info for entry: getting previous item location: unable to extract LocationRef from RepoRef
```
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
adds CI sanity test steps to github workflow
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 💻 CI/Deployment
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Internally kopia has the notion of a retention policy. This adds a pin to each snapshot that corso uses kopia to create so that those snapshots won't be subject to kopia's rentention policy. As we disable kopia's retention policy during repo init this just acts as another layer of security to ensure snapshots won't be deleted unexpectedly.
Note that this PR only applies to _new_ snapshots that corso creates.
It doesn't go back and pin older snapshots for already existing backups
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
We've found that reducing the MinEpochDuration parameter in kopia
can reduce GET requests sent to the object store by a large amount
(>50% in some cases) for repos that run many backups each day
This PR updates the init repo code path to set this parameter to
8hrs instead of the default 24hrs to bring in some of those savings
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Adds the backup's data category to the cli output.
before:
```
ID Started At Duration Status Resource Owner
09e8d1e3-92d4-426d-9ca1-3cd063ce8bb9 2024-01-12T17:54:33Z 16.103569s Completed rfinders@10rqc2.onmicrosoft.com
893ef934-9403-4775-b667-09f77cd26e83 2024-01-12T17:55:22Z 5.66253s Completed rfinders@10rqc2.onmicrosoft.com
```
after:
```
ID Started at Duration Status Protected resource Data
09e8d1e3-92d4-426d-9ca1-3cd063ce8bb9 2024-01-12T17:54:33Z 16.103569s Completed rfinders@10rqc2.onmicrosoft.com Contacts,Emails,Events
893ef934-9403-4775-b667-09f77cd26e83 2024-01-12T17:55:22Z 5.66253s Completed rfinders@10rqc2.onmicrosoft.com Emails
```
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3748
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/5046
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
Known issue, tracking in https://github.com/alcionai/corso/issues/4772
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Minor optimization where we skip trying to find BackupBases if we know
we're going to end up dropping them anyway. This can save us from
loading all the kopia manifests only to discard the information we
pulled from them. Note that this optimization only kicks in for
backups that request no merge bases and no assist bases
("full backup")
Existing test coverage in `operations/manifests_test.go` ensures this
doesn't break base selection altogether
As this is an optimization we don't need to merge this PR if it seems
like it will make code maintenance more difficult going forward
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
* #<issue>
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
enables sharepoint lists `backup` and `export`.
`restore` is enabled via `--allow-lists-restore` flag and is hidden in
CLI. So that it can be sanity-tested.
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
* Expand inReplyTo property to additionally get the parent post contents. This will be persisted as part of post data.
* This additional data will be useful for building a reply tree if we decide to do in-order restore/export in future.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Use unique info for the kopia snapshot source information to help
isolate snapshots for different resources from each other. This will
come in handy both for checkpoint snapshots and regular snapshots if
kopia's retention policy happens to look for things.
Integration tests for this probably won't pass until the next PR in
the series is merged into this PR
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
merge after:
* #5091
* #5092
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Will be used in later PRs to help make long-running streamstore
snapshots safer to create if they ever go past the kopia checkpoint
interval
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
merge after:
* #5091
#### Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Will also be used more in later PRs
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
reverses the drive backup process toggle by making the tree-based backup standard, and the old, linear enumeration the toggleable fallback.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #4690
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Switch from the `custom-query-params` kopia branch in the alcion org to
the `corsostaging` kopia branch. The `corsostaging` branch was just
updated to include recent changes from the upstream kopia repo. This
allows us to stop using our custom fork of minio because upstream kopia
has updated to the version that includes the change we need
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
generates test lists for CI sanity tests
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
adds local sanity tests for lists
for reference:
- `lists` do not have `folders` concept, they are directly created in a site. As a result cannot be restored to a different folder to test. Therefore they need to be created in adjacent to the 'backed-up' lists. Due to this there is a requirement of `SANITY_TEST_RESTORE_CONTAINER_PREFIX`. Using this, during restore check, this env-var is used to find restored lists.
- Similarly, `SANITY_TEST_SOURCE_CONTAINER` is also used a prefix matcher to filter created test lists
- When exported, lists are stored in local into the provided directory as `<SANITY_TEST_RESTORE_CONTAINER>/Lists/<list-id>/<list-id>.json`
should go in after:
- https://github.com/alcionai/corso/pull/5024
- https://github.com/alcionai/corso/pull/5026
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🤖 Supportability/Tests
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
- includes GetList in list restore handler
- includes GetListsByCollisionKey in list restore handler
- updates mock list restore handler
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
#4754
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
