## Description
Adds a func in graphConnector that reduces
siteIDs and webURLs into a set of siteIDs. This
will be used by callers such as the CLI to
generate id-based selectors for sites even if
they handle webURLs as an alternative id.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
## Description
Missed configuring working dir for staticcheck, plus fixes for errors reported.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add version of the current binary when reporting.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1594
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.154 to 1.44.155.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.155 (2022-12-07)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/ce</code>: Updates service API and documentation</li>
<li><code>service/cloudfront</code>: Updates service API and documentation
<ul>
<li>Introducing UpdateDistributionWithStagingConfig that can be used to promote the staging configuration to the production.</li>
</ul>
</li>
<li><code>service/eks</code>: Updates service API and documentation</li>
<li><code>service/kms</code>: Updates service documentation and examples
<ul>
<li>Updated examples and exceptions for External Key Store (XKS).</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="1f86cfa30f"><code>1f86cfa</code></a> Release v1.44.155 (2022-12-07) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4656">#4656</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.154...v1.44.155">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
Do not implement the StreamModTime interface so kopia-assisted incrementals cannot be used for OneDrive.
Feature should be reenabled when the following are resolved:
* #1702
## Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#1723
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add functions to Collection and Stream interfaces that allow for getting
more information about the difference between the previous backup and
the currently in-progress one. These will allow delta token-based
incremental backups to determine how the state has evolved.
Current code does not use these functions and return values for them
are "default" values that should result in full backups even if
KopiaWrapper is updated to start checking the values and GraphConnector
still pulls all items
These functions are not used during restore and can return "default"
values
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1700
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds an in-memory collection type that can be used to pass metadata files from GraphConnector to KopiaWrapper. Meant for only small amounts of data as everything must be buffered in-memory in the collection
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1685
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
This fixes#999 by skipping restore of reference attachments that are marked inline.
The attachment links (e.g. OneDrive links) are part of the message body and do not need to be
recreated.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #999
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Corso needs to store some transient information between backups so that it can perform more efficient incremental backups. This transient information will live at a path `tenant-id/service/user-id/categoryMetadata`. This patch adds an initializer, category enums, and tests for these new path locations
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1685
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Bumps [golang.org/x/tools](https://github.com/golang/tools) from 0.3.0 to 0.4.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/golang/tools/releases">golang.org/x/tools's releases</a>.</em></p>
<blockquote>
<h2>gopls/v0.4.0</h2>
<ul>
<li>Improved support for working with modules (<a href="https://github.com/ridersofrohan"><code>@ridersofrohan</code></a>). A detailed walk-through of the new features can be found <a href="https://www.ardanlabs.com/blog/2020/04/modules-05-gopls-improvements.html">here</a>. A quick summary:
<ul>
<li>Use the <code>-modfile</code> flag to suggest which modules should be added/removed from the <code>go.mod</code> file, rather than editing it automatically.</li>
<li>Suggest dependency upgrades in-editor and provide additional language features, such as formatting, for the <code>go.mod</code> file.</li>
</ul>
</li>
<li>Inverse implementations (<a href="https://github.com/muirdm"><code>@muirdm</code></a>). "Go to implementations" on a concrete type will show the interfaces it implements.</li>
<li>Completion improvements (<a href="https://github.com/muirdm"><code>@muirdm</code></a>). Specifically, improved completion for keywords. Also, offer <code>if err != nil { return err }</code> as a completion item.</li>
<li>Jumping to definition on an import statement returns all files as definition locations (<a href="https://github.com/danishprakash"><code>@danishprakash</code></a>).</li>
<li>Support for running <code>go generate</code> through the editor, via a code lens (<a href="https://github.com/marwan-at-work"><code>@marwan-at-work</code></a>).</li>
<li>Command-line support for workspace symbols (<a href="https://github.com/daisuzu"><code>@daisuzu</code></a>).</li>
</ul>
<h4>Opt-in:</h4>
<ul>
<li>Code actions suggesting <code>gofmt -s</code>-style simplifications (<a href="https://github.com/ridersofrohan"><code>@ridersofrohan</code></a>). To get these on-save, add the following setting:</li>
</ul>
<pre lang="js"><code>"[go]": {
"editor.codeActionsOnSave": {
"source.fixAll": true,
}
}
</code></pre>
<ul>
<li>Code actions suggesting fixes for type errors, such as missing return values (<a href="https://github.com/sqs/goreturns">goreturns</a>-style), undeclared names, unused parameters, and assignment statements that should be converted from <code>:=</code> to <code>=</code> (<a href="https://github.com/ridersofrohan"><code>@ridersofrohan</code></a>). Add the following to your gopls settings to opt-in to these analyzers. In the future, they will be on by default and high-confidence suggested fixes may be applied on save. See additional documentation on analyzers <a href="https://cs.opensource.google/go/tools/+/master:gopls/doc/analyzers.md">here</a>.</li>
</ul>
<pre lang="json5"><code>"gopls": {
"analyses": {
"fillreturns": true,
"undeclaredname": true,
"unusedparams": true,
"nonewvars": true,
}
}
</code></pre>
<ul>
<li>Further improvements in the support for multiple concurrent clients (<a href="https://github.com/findleyr"><code>@findleyr</code></a>). See <a href="https://github-redirect.dependabot.com/golang/tools/issues/34111">#34111</a> for all details.</li>
</ul>
<p>For a complete list of the issues resolved, see the <a href="https://github.com/golang/go/milestone/133?closed=1">gopls/v0.4.0 milestone</a>.</p>
<h2>gopls/v0.3.4</h2>
<ul>
<li><a href="https://golang.org/cl/222979">golang.org/cl/222979</a>: fix for a nil pointer exception in completion (<a href="https://github.com/muirdm"><code>@muirdm</code></a>).</li>
<li><a href="https://golang.org/cl/222980">golang.org/cl/222980</a>: fix a concurrent map write and iteration (<a href="https://github.com/ridersofrohan"><code>@ridersofrohan</code></a>).</li>
</ul>
<h2>gopls/v0.3.3</h2>
<ul>
<li>Support for workspace symbols. (<a href="https://github.com/daisuzu"><code>@daisuzu</code></a>)</li>
<li>Various completion improvements, including fixes for completion in code that doesn't parse. (<a href="https://github.com/muirdm"><code>@muirdm</code></a>)</li>
<li>Limit diagnostic concurrency, preventing huge spikes in memory usage that some users encountered. (<a href="https://github.com/heschik"><code>@heschik</code></a>)</li>
<li>Improved handling for URIs containing escaped characters. (<a href="https://github.com/heschik"><code>@heschik</code></a>)</li>
<li>Module versions from "go list" in pkg.go.dev links. (<a href="https://github.com/ridersofrohan"><code>@ridersofrohan</code></a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="aee3994bd5"><code>aee3994</code></a> gopls/internal/lsp/fake: in (*Workdir).RenameFile, fall back to read + write</li>
<li><a href="fe60148df7"><code>fe60148</code></a> go.mod: update golang.org/x dependencies</li>
<li><a href="c9ea9a72c5"><code>c9ea9a7</code></a> gopls/internal/regtest: add a test for the case when the renaming package's p...</li>
<li><a href="bf5db81001"><code>bf5db81</code></a> gopls/internal/lsp/cache: improve ad-hoc warning for nested modules</li>
<li><a href="aa9f4b2f3d"><code>aa9f4b2</code></a> go/analysis: document that facts are gob encoded in one gulp</li>
<li><a href="bdcd082252"><code>bdcd082</code></a> internal/gcimporter: skip tests earlier when 'go build' is not available</li>
<li><a href="2ad6325d90"><code>2ad6325</code></a> gopls/internal/lsp/cache: expand ImportPath!=PackagePath comment</li>
<li><a href="52c7b88fe8"><code>52c7b88</code></a> gopls/internal/robustio: only define ERROR_SHARING_VIOLATION on Windows</li>
<li><a href="4f69bf3eb3"><code>4f69bf3</code></a> gopls/internal/lsp/cache: narrow reloadOrphanedFiles to open files</li>
<li><a href="6002d6ea51"><code>6002d6e</code></a> gopls/internal/regtest/misc: test Implementations + vendor</li>
<li>Additional commits viewable in <a href="https://github.com/golang/tools/compare/v0.3.0...v0.4.0">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.153 to 1.44.154.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.154 (2022-12-06)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/billingconductor</code>: Updates service API and documentation</li>
<li><code>service/connect</code>: Updates service API, documentation, and paginators</li>
<li><code>service/dynamodb</code>: Updates service API, documentation, waiters, paginators, and examples
<ul>
<li>Endpoint Ruleset update: Use http instead of https for the "local" region.</li>
</ul>
</li>
<li><code>service/rds</code>: Updates service API, documentation, waiters, paginators, and examples
<ul>
<li>This release adds the BlueGreenDeploymentNotFoundFault to the AddTagsToResource, ListTagsForResource, and RemoveTagsFromResource operations.</li>
</ul>
</li>
<li><code>service/sagemaker-featurestore-runtime</code>: Updates service API and documentation</li>
<li><code>service/streams.dynamodb</code>: Adds new service</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="18d863d564"><code>18d863d</code></a> Release v1.44.154 (2022-12-06) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4654">#4654</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.153...v1.44.154">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
When fetching Exchange items, return the resulting delta token from the query so it can be reused later.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1685
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
OneDrive Test experienced different results locally than in CI. Test to be ignored until a consistent test can replace it.
<!-- Insert PR description-->
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
related to * #1688<issue>
## Test Plan
- [x] ⚡ Unit test
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.151 to 1.44.153.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.153 (2022-12-05)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/ce</code>: Updates service API and documentation</li>
<li><code>service/ec2</code>: Updates service documentation
<ul>
<li>Documentation updates for EC2.</li>
</ul>
</li>
<li><code>service/ivschat</code>: Updates service API and documentation</li>
<li><code>service/rds</code>: Updates service API, documentation, waiters, paginators, and examples
<ul>
<li>This release adds the InvalidDBInstanceStateFault to the RestoreDBClusterFromSnapshot operation.</li>
</ul>
</li>
<li><code>service/transcribe</code>: Updates service API</li>
</ul>
<h1>Release v1.44.152 (2022-12-02)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/appsync</code>: Updates service documentation</li>
<li><code>service/ecs</code>: Updates service documentation
<ul>
<li>Documentation updates for Amazon ECS</li>
</ul>
</li>
<li><code>service/fms</code>: Updates service API</li>
<li><code>service/mediaconvert</code>: Updates service API and documentation
<ul>
<li>The AWS Elemental MediaConvert SDK has added support for configurable ID3 eMSG box attributes and the ability to signal them with InbandEventStream tags in DASH and CMAF outputs.</li>
</ul>
</li>
<li><code>service/medialive</code>: Updates service API and documentation
<ul>
<li>Updates to Event Signaling and Management (ESAM) API and documentation.</li>
</ul>
</li>
<li><code>service/polly</code>: Updates service API
<ul>
<li>Add language code for Finnish (fi-FI)</li>
</ul>
</li>
<li><code>service/proton</code>: Updates service API</li>
<li><code>service/redshift-serverless</code>: Updates service API, documentation, and paginators</li>
<li><code>service/sns</code>: Updates service documentation
<ul>
<li>This release adds the message payload-filtering feature to the SNS Subscribe, SetSubscriptionAttributes, and GetSubscriptionAttributes API actions</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="700263c473"><code>700263c</code></a> Release v1.44.153 (2022-12-05) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4653">#4653</a>)</li>
<li><a href="6224bdd3e4"><code>6224bdd</code></a> Release v1.44.152 (2022-12-02) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4652">#4652</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.151...v1.44.153">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
Having a map of interface values instead of concrete structs allows adding collection implementations via different structs. This will be useful to inject a new collection that contains metadata information like delta links.
Also refactors the code for creating collections in Exchange so that it follows golang patterns of exiting early a bit better.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1685
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds restore commands to the cli for sharepoint.
The restore process is only partially functional at this time.
Library files that pass auth are able to be restored as expected.
However, auth issues (not directly related to these changes) prevent
restoration of all library items at this time.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1615
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Move user discovery into a `discovery` package. Also use that to return user display name from `m365.Users()` call.
This will also allow us to remove `Users` from the GC struct going forward.
If this pattern works - will follow up with a similar change for `Sites()`.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Helper function only deals with string values in the map. Returns empty string if the requested item was not present.
Will also be useful for retrieving delta tokens in future patches.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1685
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
the new pathFilterFactory now properly cleans
the inputs and checks for all-pass and all-fail
conditions.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.145 to 1.44.151.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.151 (2022-12-01)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/comprehend</code>: Updates service API, documentation, and paginators</li>
<li><code>service/gamelift</code>: Updates service API, documentation, and paginators
<ul>
<li>GameLift introduces a new feature, GameLift Anywhere. GameLift Anywhere allows you to integrate your own compute resources with GameLift. You can also use GameLift Anywhere to iteratively test your game servers without uploading the build to GameLift for every iteration.</li>
</ul>
</li>
<li><code>service/pipes</code>: Updates service API, documentation, paginators, and examples</li>
<li><code>service/states</code>: Updates service API, documentation, and paginators
<ul>
<li>This release adds support for the AWS Step Functions Map state in Distributed mode. The changes include a new MapRun resource and several new and modified APIs.</li>
</ul>
</li>
</ul>
<h1>Release v1.44.150 (2022-11-30)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/accessanalyzer</code>: Updates service API and documentation</li>
<li><code>service/athena</code>: Updates service API, documentation, and paginators
<ul>
<li>This release includes support for using Apache Spark in Amazon Athena.</li>
</ul>
</li>
<li><code>service/dataexchange</code>: Updates service API and documentation</li>
<li><code>service/docdb-elastic</code>: Adds new service</li>
<li><code>service/glue</code>: Updates service API, documentation, and paginators
<ul>
<li>This release adds support for AWS Glue Data Quality, which helps you evaluate and monitor the quality of your data and includes the API for creating, deleting, or updating data quality rulesets, runs and evaluations.</li>
</ul>
</li>
<li><code>service/s3control</code>: Updates service API and documentation
<ul>
<li>Amazon S3 now supports cross-account access points. S3 bucket owners can now allow trusted AWS accounts to create access points associated with their bucket.</li>
</ul>
</li>
<li><code>service/sagemaker</code>: Updates service API, documentation, and paginators
<ul>
<li>Added Models as part of the Search API. Added Model shadow deployments in realtime inference, and shadow testing in managed inference. Added support for shared spaces, geospatial APIs, Model Cards, AutoMLJobStep in pipelines, Git repositories on user profiles and domains, Model sharing in Jumpstart.</li>
</ul>
</li>
<li><code>service/sagemaker-geospatial</code>: Adds new service</li>
</ul>
<h1>Release v1.44.149 (2022-11-29)</h1>
<h1>Release v1.44.148 (2022-11-29)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/ec2</code>: Updates service API, documentation, and paginators
<ul>
<li>This release adds support for AWS Verified Access and the Hpc6id Amazon EC2 compute optimized instance type, which features 3rd generation Intel Xeon Scalable processors.</li>
<li>This release adds support for AWS Verified Access and the Hpc6id Amazon EC2 compute optimized instance type, which features 3rd generation Intel Xeon Scalable processors.</li>
</ul>
</li>
<li><code>service/firehose</code>: Updates service API and documentation
<ul>
<li>Allow support for the Serverless offering for Amazon OpenSearch Service as a Kinesis Data Firehose delivery destination.</li>
<li>Allow support for the Serverless offering for Amazon OpenSearch Service as a Kinesis Data Firehose delivery destination.</li>
</ul>
</li>
<li><code>service/kms</code>: Updates service API, documentation, paginators, and examples
<ul>
<li>AWS KMS introduces the External Key Store (XKS), a new feature for customers who want to protect their data with encryption keys stored in an external key management system under their control.</li>
<li>AWS KMS introduces the External Key Store (XKS), a new feature for customers who want to protect their data with encryption keys stored in an external key management system under their control.</li>
</ul>
</li>
<li><code>service/omics</code>: Updates service API, documentation, waiters, paginators, and examples</li>
<li><code>service/opensearchserverless</code>: Adds new service</li>
<li><code>service/securitylake</code>: Adds new service</li>
<li><code>service/simspaceweaver</code>: Adds new service</li>
</ul>
<p>Release v1.44.147 (2022-11-29)</p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="9c070a47ed"><code>9c070a4</code></a> Release v1.44.151 (2022-12-01) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4650">#4650</a>)</li>
<li><a href="9546abe01a"><code>9546abe</code></a> Release v1.44.150 (2022-11-30) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4648">#4648</a>)</li>
<li><a href="bf389b3c58"><code>bf389b3</code></a> Release v1.44.149 (2022-11-29) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4647">#4647</a>)</li>
<li><a href="9e891321fb"><code>9e89132</code></a> Fix KMS example models. (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4646">#4646</a>)</li>
<li><a href="b4def0753f"><code>b4def07</code></a> Release v1.44.148 (2022-11-29) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4644">#4644</a>)</li>
<li><a href="4133cb1ae7"><code>4133cb1</code></a> Release v1.44.147 (2022-11-29) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4639">#4639</a>)</li>
<li><a href="d9261b2f95"><code>d9261b2</code></a> Release v1.44.146 (2022-11-28) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4638">#4638</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.145...v1.44.151">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
Add ModTime to Exchange, OneDrive and SharePoint list stream items. This enables kopia-assisted incrementals for those items. Backup details still contains a complete set of information for all items in the backup regardless of if kopia uploaded data for the item or not.
Kopia-assisted incrementals does come with some caveats though. If changes are made to an item in M365 and that change does not cause the modified time reported by M365 to update, then the change will not be backed up. Currently, only marking an email as read/unread is known to hit this edge case.
This patch does not lazily fetch data from Graph API. This means that kopia may upload less data, but the same amount of data will still be pulled from Graph
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#622
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds webUrl scopes to the sharepoint selector.
Also introduces the idea of a scope categories
that can broadly match across all leaf types
within a service. This category union should be
reserved for root categories, and properties
that can be used interchangably with the root.
This is part 2 of exposing webURLs as an alternative
to siteIDs for sharepoint backup and restore. The
next change providing a webURL => siteID lookup
within the graph package.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
## Description
Kiota serialization patch found located `exchange_data_collection.go`L 316 required a large collection of functions to make the message object serializable. Kiota upgrades since instantiation no longer require this collection of functions.
Dead code removed from m365Transform.go
<!-- Insert PR description-->
## Type of change
- [x] 🗺️ Documentation
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Will fit better with how incremental backups are planned to work in the future as next links can be extracted.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
part of:
* closes#1612
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Logic for List retrieval from M365 back store. Keeping separate from Collection creation logic as there are a lot of calls to retrieve all supporting relationship data from the backstore.
Adds collection to
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1475<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Set the mod time of uploaded files to either the mod time of the item (if it has one) or the current time (if it does not have one). Also add tests to check caching in kopia works properly.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#621
part of:
* #547
merge after:
* #1427
* #1430
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Cleanup all folders instead of just some of them. Log errors instead of failing test
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#1661
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Tell kopia about previous snapshots during the backup
so it can make use of them to skip uploading some data.
Currently only selects the most recent completed snapshot
and most recent incomplete snapshot that contains the
information being backed up. This can be tuned later if
it is not working good enough. However, increasing the
number of previous snapshots passed in may increase memory
usage during backup
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1404
merge after:
* #1427
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add size and stored size to cli output for `backup list`
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/1622
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Use delta endpoint to get contacts in each folder
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
part of:
* #1612
merge after:
* #1620
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Preparing for handling sharepoint webUrl
selectors, we will need suffix-matching filters.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1616
## Test Plan
- [x] ⚡ Unit test
## Description
Use delta endpoint to fetch exchange mail items. Disables picking only some item attributes as query parameters are not implemented in this version of Graph SDK.
Switches to custom pagination iteration method so that next links and delta tokens can be extracted later on.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1612
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Minor fixes for flakey tests:
* Dedupe OneDrive folders by checking IDs. Should help remove some test flakes.
* Weaken checks on kopia snapshot manager tests as they use maps with indeterminate iteration orders
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1660
* closes#1657
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Helper functions that allow finding the most recent complete
(and maybe incomplete) snapshots for a given set of tags. The
returned snapshots can later be used to do more efficient kopia
snapshots by allowing kopia to determine it's already uploaded
a file and skip uploading it again.
The number of most recent snapshots to return can be tuned (right
now it returns 1 most recent), but may cause more memory usage
during backups.
Kopia currently has some oddities when getting snapshot
manifests via tags:
* tag values cannot be empty or the comparison returns always
true, selecting all snapshot manifests
* kopia does not currently tag snapshot manifests made during
checkpoints. A patch to upstream kopia is needed to fix this.
All other manifests that match the tags will be returned though
Added code is not currently connected to any backup logic.
Viewing by commit may help as there was a tad of lift'n'shift code
movement for the sake of organization
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1404
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
This improves the initial backup speed for OneDrive. OneDrive backup was mostly slow when we had a lot of tiny files. Case where we had mostly large files was pretty much the best case scenario and we were throttled by purely how fast we can get the files from MS and how fast kopia can process and upload it. But of small files, we were slowed by the loop which was taking quite a bit of time to fetch the download urls. We have now parallelized the query for getting the download URL. Under best case scenarios, I was able to speed it up to under 20s from ~4-5m starting point. That said, MS graph api still seems to throttle us and when that happen we still go back to around ~2m for worst case scenario. I've added 3 retries as some requests were failing when we continuously making many requests.
This should also take care of the issue of url expiring mentioned in https://github.com/alcionai/corso/issues/581 as we are only prefetching a few urls ahed of time.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1595
* fixes https://github.com/alcionai/corso/issues/581
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
From rough numbers we can speedup an account with ~3000 emails, ~1000 contacts and ~1000 events from ~18m to <3m.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/362
* https://github.com/alcionai/corso/issues/1595
* https://github.com/alcionai/corso/pull/1607
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Moves the tenantID out of the repoID hash and
into its own value (also hashed) in the event log.
Unique user identification to follow.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1602
## Test Plan
- [x] 💚 E2E
## Description
Sketch of how to split backups by (resource owner, service) for exchange. Needs more support from selectors package to be complete.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1505
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Use latest golangci-lint version. Also updates cache flags.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1629
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
The current version of graph SDK (v0.40.0) Corso uses does not support adding query parameters to delta requests on exchange mail folder messages or contact folder contacts. This reimplements the logic from a newer version of graph SDK so that we can add those parameters. Much of the code is lifted from the updated version of the SDK, with some combining of types that have the same fields.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1612
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Helper function to fetch backup models by ID. Also renames `Backups` to `BackupsByTag`
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1505
separated from
* #1609
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds the `backup details sharepoint` command
to the CLI. E2E support is already functional.
Additional filters will be added as needs are
identified.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1614
## Test Plan
- [x] ⚡ Unit test
## Description
Add a Set(ctx, logger) func to the logger package
to allow SDK users to provide their own zap
logger instance to corso.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1630
## Description
Adds a method to all scopes that produces a
discrete clone of the original scope. The result is a shallow clone of the scope, thus retaining all
original values, but is made discrete by replacing the resource owner target with a specific value.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1621
## Test Plan
- [x] ⚡ Unit test
