## Description
If an exchange collection only has previous
delta info, or only previous path info, and not
the other, remove that entry from the parsed
metadata map, causing a full backup of the
containere.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1804
## Test Plan
- [x] ⚡ Unit test
## Description
Updating the isRead flag in M365 does not cause the mod time of the item to be updated. This means delta queries will not return items marked as [un]read when isRead is not one of the attributes requested in the $select clause.
Explicitly fetch isRead so that we get those items. Further changes will be required to ensure the mod time of the item is something that will not cause kopia-assisted dedupe to skip the item when only the isRead attribute has been changed.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1866
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Upstream Breaking changes applied for upgrade to the following libraries.
- kiota-abstractions-go from 0.14.0 to 0.15.1
- msgraph-sdk-go from 0.48.0 to 0.49.0
- msgraph-sdk-go-core from 0.30.1 to 0.31.0
- kiota-http-go from 0.9.0 to 0.10.0
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor --> Library support and Upstream bug fixes
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#1856 <issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Have BackupOp produce the paths for relevant subtrees in each snapshot and pass those to BackupCollections. This removes the need for code in the kopia package to call into more service/category-specific path package code, thus keeping the kopia package more generic.
As in #1828, prefix info for each subtree path is pulled from the Reason a snapshot was selected.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1832
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.161 to 1.44.162.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.162 (2022-12-16)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/appflow</code>: Updates service API and documentation</li>
<li><code>service/cloudfront</code>: Updates service documentation
<ul>
<li>Updated documentation for CloudFront</li>
</ul>
</li>
<li><code>service/datasync</code>: Updates service API and documentation</li>
<li><code>service/elasticfilesystem</code>: Updates service documentation
<ul>
<li>General documentation updates for EFS.</li>
</ul>
</li>
<li><code>service/guardduty</code>: Updates service documentation
<ul>
<li>This release provides the valid characters for the Description and Name field.</li>
</ul>
</li>
<li><code>service/iotfleetwise</code>: Updates service API</li>
<li><code>service/sagemaker</code>: Updates service API and documentation
<ul>
<li>AWS sagemaker - Features: This release adds support for random seed, it's an integer value used to initialize a pseudo-random number generator. Setting a random seed will allow the hyperparameter tuning search strategies to produce more consistent configurations for the same tuning job.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="5afd13d6ea"><code>5afd13d</code></a> Release v1.44.162 (2022-12-16) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4666">#4666</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.161...v1.44.162">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
Fulfills the data collection interfaces to provide item deleted
flags and collection state enums.
A note on the additional data consumption: I'm making an assumption
that the @removed property will appear in that map. This is awaiting
testing for verification, which I'll get into as a follow-up step.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1727
## Test Plan
- [x] ❓ In a follow-up PR
## Description
During BackupCollections, use the base snapshots that were passed in
from the BackupOp in the kopia uploader instead of fetching our own set
of base snapshots. This will ensure the set of base snapshots is
consistent throughout the backup.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1740
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
For tests that run subtests, make sure to use the subtest instance of testing.T so that the error trace in the output is easier to read.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#1846
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Update the `FolderEntry` type with placeholder fields for folder size and modified time values.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1850
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Use the Reasons a snapshot was selected to retrieve only the metadata
corresponding to those reasons. This will avoid having multiple versions
of metadata for the same (resource owner, service, category) tuple as
well as pulling in more metadata than required for some backups.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1829
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
When backing up Exchange, don't return an error if the folder/calendar we're trying to fetch item IDs for has been deleted.
Error codes pulled with graph explorer API
Manually tested with an Exchange mail backup
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1846
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
These two structures are effectively identical.
This refactor deferrs to the graph version since
services may still wrap that for more specific
behavior as needed.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1727
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Pick what data to source from a base snapshot by examining the reason the snapshot was selected as a base. This helps avoid two issues:
* pulling in unwanted data when the base has a superset of what is being backed up. Example: pulling in contacts from the base when only email is being backed up
* clobbering already selected data from a different base with data in the base currently being examined. Example: two snapshots, one with contacts and emails and the other with just emails. Second snapshot is newer than the first. The email items in the first snapshot should not clobber those in the first when building the hierarchy
This PR also has the effect of, under some conditions, reducing the amount of data that is pulled from the remote kopia repo when building the hierarchy. This occurs because only the subtrees that will be used in the new backup are traversed instead of traversing the entire snapshot
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1740
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Merge directory layouts between the passed in collections and the base snapshot(s). Also add unit tests to ensure the output kopia hierarchy looks as expected. (CLI) user observable behavior is not affected by this PR
This PR does not address:
* selecting subtrees for specific data categories in base snapshots
* not clobbering more recent info if multiple snapshots have subtrees for the same data category (ties into above)
* file deletions for services that can only report item deletions at a global level (e.x. OneDrive file deletions)
Viewing individual commits in PR may make changes easier to review
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1740
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Fixes warnings the H1 headers are missing
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
- [x] 🗺️ Documentation
## Description
Parses the previousPaths metadata collections
along with deltas, and hands paths down to
exchange backup collection producers. Does
not yet scrutinize previous/current path diffs.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1726
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Reverts alcionai/corso#1835 now that #1841 is in. That should help with the flakiness.
I will follow up with another PR after though that moves these tests to a post-CI job (we don't need to run these on every commit)
## Description
Adds an additional metadata collection: a folder
id to path string mapping. This collection is
created on backup, and retrieved along with
the delta metadata on the next backup, but is
not yet parsed or utilzed downstream.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1726
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Initial changes to support SharePoint Lists being chosen for Backup Operations.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
* close #1785<issue>
## Test Plan
- [x] ⚡ Unit test
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.160 to 1.44.161.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.161 (2022-12-15)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/backup-gateway</code>: Updates service API and documentation</li>
<li><code>service/connect</code>: Updates service API and documentation</li>
<li><code>service/ecs</code>: Updates service API and documentation
<ul>
<li>This release adds support for container port ranges in ECS, a new capability that allows customers to provide container port ranges to simplify use cases where multiple ports are in use in a container. This release updates TaskDefinition mutation APIs and the Task description APIs.</li>
</ul>
</li>
<li><code>service/eks</code>: Updates service API and documentation</li>
<li><code>service/glue</code>: Updates service API and documentation
<ul>
<li>This release adds support for AWS Glue Crawler with native DeltaLake tables, allowing Crawlers to classify Delta Lake format tables and catalog them for query engines to query against.</li>
</ul>
</li>
<li><code>service/kinesis</code>: Updates service API, documentation, and paginators
<ul>
<li>Added StreamARN parameter for Kinesis Data Streams APIs. Added a new opaque pagination token for ListStreams. SDKs will auto-generate Account Endpoint when accessing Kinesis Data Streams.</li>
</ul>
</li>
<li><code>service/location</code>: Updates service documentation</li>
<li><code>service/m2</code>: Updates service API and documentation</li>
<li><code>service/sagemaker</code>: Updates service API and documentation
<ul>
<li>SageMaker Inference Recommender now allows customers to load tests their models on various instance types using private VPC.</li>
</ul>
</li>
<li><code>service/securityhub</code>: Updates service API and documentation</li>
<li><code>service/translate</code>: Updates service API and documentation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="a9faefa96c"><code>a9faefa</code></a> Release v1.44.161 (2022-12-15) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4665">#4665</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.160...v1.44.161">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
These tests have the highes rate of timeout
based failures, while providing the most loose
gurantees about system behavior. This will
temporarily disable them while we consider
separating our suite between integration
and e2e smoke sets.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🤖 Test
## Test Plan
- [x] 💚 E2E
## Description
Fix issues found by screaming frog
- Use https:// for links
- Add missing cookies page back
- Add missing alt-text
- Use dashes instead of underscores for better search engine indexing
- Fix internal redirects
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Description
Fixes two bugs: 1. building the correc tag key
when retrieving the prior backup id from a
manifest. 2. returning ErrNotFound when a
snapshot entry isn't found for restore. This
can occur for metadata if the service has
a prior backup that does not contain the
expected metadata files.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1823
## Test Plan
- [x] 💪 Manual
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.159 to 1.44.160.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.160 (2022-12-14)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/ce</code>: Updates service API and documentation</li>
<li><code>service/monitoring</code>: Updates service API and documentation
<ul>
<li>Adding support for Metrics Insights Alarms</li>
</ul>
</li>
<li><code>service/networkmanager</code>: Updates service API and documentation</li>
<li><code>service/redshift-data</code>: Updates service API and documentation</li>
<li><code>service/sagemaker-metrics</code>: Updates service documentation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="d7b4743f70"><code>d7b4743</code></a> Release v1.44.160 (2022-12-14) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4663">#4663</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.159...v1.44.160">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
Adds a flag in backup details that indicates whether the item in that backup is new/updated.
Currently always set to `true` but once we implement #1800 - we will set this to false for existing items from
base snapshots.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1812
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Create and wire up a helper function that merges items in the base snapshot with items that have been seen from the collection
Will not change the output of execution at all since the base snapshot directory will always be nil at the moment
Will require further modifications to properly handle deleted items where only a single delta endpoint is used to fetch changes (e.x. OneDrive file deletions)
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1740
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add a helper function that allows loading the root directory of a snapshot given the snapshot manifest. This will allow future code for merging directory hierarchies to be tested without having to involve an actual instance of kopia with some previously created snapshots.
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1740
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
For each snapshot manifest returned when looking for previous snapshots for a given set of owners cats, return the reason the snapshot was selected. This will allow other code to select the correct metadata and base snapshot subtree(s) when making incremental backups.
An example of when all metadata and directories in a base snapshot may not be needed is
```text
backup create exchange --data email,contacts --users user1 -> B1
// uses B1 as the base
backup create exchange --data email --users user1 -> B2
// uses B1 as the base for contacts and B2 as the base for email
backup create exchange --data email,contacts --users user1 -> B3
```
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1779
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.158 to 1.44.159.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.159 (2022-12-13)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/cloudtrail</code>: Updates service API and documentation
<ul>
<li>Merging mainline branch for service model into mainline release branch. There are no new APIs.</li>
</ul>
</li>
<li><code>service/rds</code>: Updates service API, documentation, waiters, paginators, and examples
<ul>
<li>This deployment adds ClientPasswordAuthType field to the Auth structure of the DBProxy.</li>
</ul>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="183764bff4"><code>183764b</code></a> Release v1.44.159 (2022-12-13) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4661">#4661</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.158...v1.44.159">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
Instead of relying on KopiaWrapper to create the OwnersCats for a
backup, have BackupOp create them from the selector and pass them in.
This is necessary as incremental backups will no longer see all the data
in the backup, meaning it cannot accurately create the OwnersCats
because some data categories or owners in the backup may not have had
changes.
OwnersCats are eventually converted to tags on a kopia snapshot and used
to lookup snapshots when trying to find base snapshots for incrementals.
Additional minor changes:
* use pointers instead of values when passing parameters
* set backup details OwnersCats to nil
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1781
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Relocates the graphService struct to graph as
the Service struct. Replaces GC's embedded
graphService with a graph.Servicer reference.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1725
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Extends the columns printed on sharePoint details
to include the expected data. Fixes some bugs with
flag processing for details and restore commands.
## Type of change
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
* #1616
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Configuration and attenion to the graphService
failFast is haphazard and has shared ownership.
This change removes that property from the
service, along with the ErrPolicy func, in favor of passing around a control.Options struct.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1725
* #302
## Test Plan
- [x] ⚡ Unit test
## Description
Update mockcollection to allow returning non-default values for attributes like PreviousPath() and whether an item has been deleted.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1740
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
When backing up exchange data, parse the
metadata collection of delta urls from prior runs
(if any exist) and pass those tokens along to the
fetch functions for re-use.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1725
## Test Plan
- [x] ⚡ Unit test
## Description
Addition of SharePoint-specific functions to test package. Verifies that support package functions are present and do not cause errors upon SharePoint Collection Creation
## Type of change
- [x] 🐹 Trivial/Minor
## Test Plan
- [x] ⚡ Unit test
## Description
`graph.Service -> graph.Servicer`, no other changes.
More compliant with golang naming standards,
and will allow us to eventually migrate the
Service struct out of connector and into graph.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1725
## Description
Supporting libraries updated:
- github.com/cjlapao/common-go v0.0.36
- golang.org/x/net v0.4.0
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
Related to #1479
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.157 to 1.44.158.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.158 (2022-12-12)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/customer-profiles</code>: Updates service API and documentation</li>
<li><code>service/ec2</code>: Updates service API and documentation
<ul>
<li>This release updates DescribeFpgaImages to show supported instance types of AFIs in its response.</li>
</ul>
</li>
<li><code>service/kinesisvideo</code>: Updates service API and documentation
<ul>
<li>This release adds support for public preview of Kinesis Video Stream at Edge enabling customers to provide configuration for the Kinesis Video Stream EdgeAgent running on an on-premise IoT device. Customers can now locally record from cameras and stream videos to the cloud on configured schedule.</li>
</ul>
</li>
<li><code>service/lookoutvision</code>: Updates service documentation</li>
<li><code>service/migration-hub-refactor-spaces</code>: Updates service API and documentation</li>
<li><code>service/rds</code>: Updates service API, documentation, waiters, paginators, and examples
<ul>
<li>Update the RDS API model to support copying option groups during the CopyDBSnapshot operation</li>
</ul>
</li>
<li><code>service/rekognition</code>: Updates service API and documentation
<ul>
<li>Adds support for "aliases" and "categories", inclusion and exclusion filters for labels and label categories, and aggregating labels by video segment timestamps for Stored Video Label Detection APIs.</li>
</ul>
</li>
<li><code>service/sagemaker-metrics</code>: Adds new service</li>
<li><code>service/wafv2</code>: Updates service documentation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="25ac8eb046"><code>25ac8eb</code></a> Release v1.44.158 (2022-12-12) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4660">#4660</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.157...v1.44.158">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
In a backup operation, begins the operation by
retrieving all backup manifests and metadata
from prior operations.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1725
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
generates a unique repository ID on repo init, and retrieves it again on repo connect. This allows
the event bus (and any other tracker) to id a repo independent of other context.
I've been unable to find any good way to retrieve a unique repository ID from kopia. This seems like
the next quickest solution.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1602
## Test Plan
- [x] 💪 Manual
## Description
Instead of rolling our own logic for injecting a version, use a
MultiReader to concatenate the streams. Handling Close() is now more
complex though.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#1766
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Move helper functions & test structs to `helper_test.go`
<!-- Insert PR description-->
## Type of change
- [x] 🐹 Trivial/Minor
## Test Plan
- [x] ⚡ Unit test
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.155 to 1.44.157.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.157 (2022-12-09)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/iotfleetwise</code>: Updates service API and documentation</li>
<li><code>service/logs</code>: Updates service documentation
<ul>
<li>Doc-only update for CloudWatch Logs, for Tagging Permissions clarifications</li>
</ul>
</li>
<li><code>service/medialive</code>: Updates service API and documentation
<ul>
<li>Link devices now support buffer size (latency) configuration. A higher latency value means a longer delay in transmitting from the device to MediaLive, but improved resiliency. A lower latency value means a shorter delay, but less resiliency.</li>
</ul>
</li>
<li><code>service/mediapackage-vod</code>: Updates service API and documentation</li>
</ul>
<h1>Release v1.44.156 (2022-12-08)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/autoscaling</code>: Updates service API and documentation
<ul>
<li>Adds support for metric math for target tracking scaling policies, saving you the cost and effort of publishing a custom metric to CloudWatch. Also adds support for VPC Lattice by adding the Attach/Detach/DescribeTrafficSources APIs and a new health check type to the CreateAutoScalingGroup API.</li>
</ul>
</li>
<li><code>service/iottwinmaker</code>: Updates service API, documentation, and paginators</li>
<li><code>service/migrationhubstrategy</code>: Updates service API and documentation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="26b0196178"><code>26b0196</code></a> Release v1.44.157 (2022-12-09) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4658">#4658</a>)</li>
<li><a href="e6bbaf445e"><code>e6bbaf4</code></a> Release v1.44.156 (2022-12-08) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4657">#4657</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.155...v1.44.157">compare view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
You can trigger a rebase of this PR by commenting `@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
</details>
## Description
We have a retry handler within the client added as a middleware. So, don't retry here again if we still get any other err.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/1684
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
This PR uses a separate snapshot to store backup details instead of a Corso model (i.e. a kopia manifest).
Introduces a `StreamStore` that can be leveraged to store larger metadata objects. We can also leverage this
for incrementals or restartable backups going forward.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1735
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Track additional information about collections and their items during backup so we can properly merge directories and items in directories when doing incremental backups
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1740
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
