1512 Commits

Author SHA1 Message Date
Keepers
9dab7c0010
add selector, shortref logging in operations (#2605)
Some additional logging for debugging
2023-02-22 01:48:36 +00:00
Danny
f06f2f585a
GC: OneDrive/SharePoint WebURL Fix (#2596)
## Description
Changes: 
- Use of `ptr` package to ensure safe dereferencing of pointers
- Update to query to ensure that `sharepointIds` are part of the response
- Alternate method created to obtain originating sharepoint site url 
<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1747<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-02-21 23:29:31 +00:00
Keepers
a88b984b9a
log all recoverable errors in backup/restore (#2598)
## Description

fault is aggregating recoverable errors, but no code currently reports them.  This is a quick hack to add logging around those errors.  In the future, we'll want to refine who and where performs this report.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x] 💪 Manual
2023-02-21 20:57:30 +00:00
Nočnica Mellifera
d7c0f61105
new blog post on large backup best practices (#2446)
## Description

new blog post on large backup best practices

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E

---------

Co-authored-by: Niraj Tolia <ntolia@users.noreply.github.com>
2023-02-21 12:07:59 -08:00
Keepers
533164744b
standardize panic recovery (#2530)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2529

## Test Plan

- [x]  Unit test
2023-02-21 18:36:56 +00:00
ashmrtn
cfc2b2a206
Use tester suite wrapper in cli package (#2593)
## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2373

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-21 18:12:02 +00:00
ashmrtn
6e0b9183b4
OneDrive test framework (#2533)
## Description

Major changes:
* OneDrive files now store the lookup key in the first few bytes of the
  file and then the file data. This allows the test framework to
  continue working even when we switch to using item IDs as names
* Create a framework to create the data necessary for testing. This
  framework uses something akin to the builder pattern and automatically
  adds files to the collection if they should appear in that version.
  For example, trying to add a metadata file to a backup version that
  didn't have metadata files will result in a noop. Adding files will
  also add the metadata to the aux file set

Long term this will hopefully make ensuring compatibility between OneDrive
versions easier because only a few places in the code need to be updated
to support another version. It also allows easy ranges over different
versions via for-loop

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1535

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-21 16:39:48 +00:00
dependabot[bot]
19dad32db9
⬆️ Bump github.com/aws/aws-sdk-go from 1.44.204 to 1.44.205 in /src (#2594)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.204 to 1.44.205.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.205 (2023-02-20)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/auditmanager</code>: Updates service API and documentation</li>
<li><code>service/connect</code>: Updates service API and documentation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="162792e59c"><code>162792e</code></a> Release v1.44.205 (2023-02-20) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4732">#4732</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.204...v1.44.205">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/aws/aws-sdk-go&package-manager=go_modules&previous-version=1.44.204&new-version=1.44.205)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2023-02-21 11:18:53 +00:00
Abin Simon
3207263425
Add both data and meta file to exclude list (#2557)
## Description

This makes sure both the `.data` and `.meta` files for a OneDrive item are added to the exclude list so that both are deleted. It does nothing for dirs as of now as we will be moving the directory metadata to within the directory so that we can leverage kopia tree operations to take care of that.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2117

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-21 10:34:22 +00:00
Abin Simon
3b516e551f
Handle files moved multiple times within a delta query (#2560)
## Description

It is possible that within a single delta query that files could be moved multiple times, this fixes that.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2559

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-21 09:56:06 +00:00
Abin Simon
dd72c4c8f9
Mark collections as Deleted if not available and invalid delta (#2503)
## Description

<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Final entry : closes https://github.com/alcionai/corso/issues/2242

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-21 09:28:51 +00:00
Abin Simon
8b21e25a82
Mark v0.4.0 as beta (#2592)
We are now in beta! 🎊
2023-02-21 04:27:10 +00:00
ashmrtn
ed6335e590
New restore path for items with name in metadata file (#2495)
## Description

Add code to restore items that have their file name stored in the corso metadata file instead of as the name of the kopia file.

This code is not executed as the conditional for it will not trigger (comparison to max int)

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1535 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-21 04:26:25 +00:00
ashmrtn
4565c9f33b
Begin storing OneDrive file names in metadata (#2494)
## Description

This starts to store the name of the OneDrive file/directory in the corso metadata file for the OneDrive file/directory. This data is not yet used anywhere

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1535

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-21 03:12:16 +00:00
Keepers
d7087d7461
add awareness of missing folders (#2578)
## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🗺️ Documentation
## Issue(s)

* #2567
2023-02-20 22:22:40 +00:00
Danny
7ba3be5242
CLI: SharePoint --data flag and wildcards (#2580)
## Description
Enables feature SharePoint to use "wildcard" to select all available sites with the ability to only backup selected categories.

Example 
```bash
./corso backup create sharepoint --web-url "*" --data libraries
```



<!-- Insert PR description-->

## Does this PR need a docs update or release note?
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2579<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-02-20 21:21:27 +00:00
Keepers
960cb45ca8
write test logs to stdout (#2568)
stderr logging writes all our logs to the terminal (ie, what shows up in github workflows) instead of the logging file artifact.  On failures with thousands of lines of logs, this makes searching for the error source nearly impossible.  This change ensures we have clean outputs in github workflows, and logs stored in artifact files for extensive review.
2023-02-20 19:49:50 +00:00
ashmrtn
cf62c6283c
Consolidate logic to see if Exchange/OneDrive user exists (#2508)
## Description

Use the check to see if a user has access to OneDrive/Exchange to also see if the user exists. This does not stop the population of GraphConnector users every time an instance is created, but does move us one step closer to that

Manually tested the following cases:
* backup "*"
* backup single existing user without error
* backup multiple existing users without error
* error when attempting to backup user that doesn't exist in the same domain
* error when attempting to backup user that doesn't exist in a different domain

Sample error output (progress disabled):
```text
No backups available
Error: 1 error occurred:
        * Failed to run Exchange backup for user foo@other.onmicrosoft.com: doing backup: producing backup data collections: resource owner [foo@other.onmicrosoft.com] not found within tenant
```

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1547 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-20 17:50:47 +00:00
Abin Simon
a86453f138
Add owner in json output for backup list (#2575)
## Description

This adds information about owner of the backup to `--json` output of backup list. Previously we were showing owner in the normal backup list command, but not when asked to as json.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-20 14:50:28 +00:00
Abin Simon
8096025c55
Print api response in logs for 4xx errors (#2544)
## Description

Print api response in logs for 4xx errors from GC. More info for things like 400 Bad Request could be useful.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-20 09:14:23 +00:00
neha_gupta
2b8e1ad2d9
Update CHANGELOG.md (#2574)
## Description

Updating changelog as v0.0.4 is out.

## Does this PR need a docs update or release note?
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🗺️ Documentation

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan
- [ ] 💪 Manual
2023-02-20 08:04:27 +00:00
dependabot[bot]
0c76903dc4
⬆️ Bump sass from 1.58.1 to 1.58.3 in /website (#2572) 2023-02-20 06:23:27 +00:00
dependabot[bot]
d2d2fe1197
⬆️ Bump mermaid from 9.3.0 to 9.4.0 in /website (#2571) 2023-02-20 06:18:18 +00:00
Keepers
5707036b7c
add clues/fault to sharepoint api (#2507)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
v0.4.0
2023-02-18 20:42:48 +00:00
Keepers
b1ff20d36c
add clues/fault to sharepoint collections (#2506)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-18 19:51:51 +00:00
Abin Simon
7e3532832e
More fine grained error messages for retries (#2545)
## Description

This should help in figuring out more retry failures.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-18 17:16:42 +00:00
Niraj Tolia
6b79355948
Move Corso to beta (#2566)
## Description

Move Corso out of alpha

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🗺️ Documentation
2023-02-18 15:35:59 +00:00
ashmrtn
e215462518
Add log messages about metadata saved/used (#2551)
## Description

Record the number of entries in the saved metadata. This could help catch odd situations like not finding anything to backup.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2550

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-18 06:25:05 +00:00
dependabot[bot]
bf12428738
⬆️ Bump github.com/aws/aws-sdk-go from 1.44.202 to 1.44.204 in /src (#2564)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.202 to 1.44.204.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.204 (2023-02-17)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/apprunner</code>: Updates service API</li>
<li><code>service/glue</code>: Updates service API and documentation
<ul>
<li>Release of Delta Lake Data Lake Format for Glue Studio Service</li>
</ul>
</li>
</ul>
<h1>Release v1.44.203 (2023-02-16)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/elasticmapreduce</code>: Updates service API and documentation
<ul>
<li>This release provides customers the ability to define a timeout period for procuring capacity during a resize operation for Instance Fleet clusters. Customers can specify this timeout using the ResizeSpecifications parameter supported by RunJobFlow, ModifyInstanceFleet and AddInstanceFleet APIs.</li>
</ul>
</li>
<li><code>service/grafana</code>: Updates service API and documentation</li>
<li><code>service/ivs</code>: Updates service documentation</li>
<li><code>service/wafv2</code>: Updates service documentation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="ad756cba2a"><code>ad756cb</code></a> Release v1.44.204 (2023-02-17) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4730">#4730</a>)</li>
<li><a href="8f70ea398e"><code>8f70ea3</code></a> add go 1.20 support (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4724">#4724</a>)</li>
<li><a href="ec9362abbd"><code>ec9362a</code></a> Release v1.44.203 (2023-02-16) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4727">#4727</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.202...v1.44.204">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/aws/aws-sdk-go&package-manager=go_modules&previous-version=1.44.202&new-version=1.44.204)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2023-02-18 05:46:46 +00:00
Vaibhav Kamra
c3f5e50b9f
Fix how ErrrorData is marshaled (#2565)
## Description

The error type does not get marshaled which can lead to errors when we try to un-marshal a manifest
that had errors stored in it.

Repro here: https://go.dev/play/p/tgj8oq5CGFd

For now - this disables JSON marshaling and also fixes the error un-marshaling. I have added a regression test
to verify both behaviors.

As a follow-up, I believe we can implement a custom marshaler.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-18 01:43:59 +00:00
Keepers
207232e8d9
add clues/fault to sharepoint restores (#2505)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-17 22:05:32 +00:00
Vaibhav Kamra
daad056d7e
Pull in fix for concurrent map updates in clues (#2562)
## Description

Pulls in fix for a panic caused by concurrent map updates in the clues pkg

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-02-17 21:24:06 +00:00
Danny
62d652764a
GC: Ptr clean-up and Invalid NextLink logging (#2547)
## Description
- Use of ptr package for next link and delta links. 
- Added logging to verify the presence of receiving invalid next links from M365.


<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  No 

- Related to #2520
## Type of change
- [x] 🧹 Tech Debt/Cleanup

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x]  Unit test
2023-02-17 19:20:21 +00:00
Keepers
d395311821
error logging around malformed urls (#2561)
adds debug logging around url construction
2023-02-17 18:16:45 +00:00
Keepers
c92b70e000
wrap up clues and fault additions to exchange (#2504)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-17 17:08:08 +00:00
Niraj Tolia
7c8c464fea
Fix Corso panics in Docker images (#2543)
## Description

The `KOPIA_CACHE_DIRECTORY` env variable causes Corso to panic even if `$CORSO_HOME/kopia/cache` exists.
Removing it fixes the builds but I am unsure of any side effects.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #2529 

## Test Plan

- [x] 💪 Manual
2023-02-17 05:29:58 +00:00
ashmrtn
0ff218ab3e
Hook into error callback in kopia (#2555)
## Description

Using this over extracting errors from the snapshot allows us to see all errors. The errors that each directory tracks are truncated when the directory is serialized for persistence if there're too many.

With the current `fault.Errors` behavior this cases the backup to fail if there's any errors
kopia ignored. This is consistent with behavior in the backup op that checks for any errors.

Unclear what the CLI output and log messages will be for these errors as they are grouped
into all other errors reported through the fault package

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2548

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-17 04:32:44 +00:00
ashmrtn
7b9ba935e8
log and continue if kopia can't find expected base snapshot directory (#2552)
## Description

This allows corso to continue making a snapshot even if the expected directory in the base is not found. The fact that the entry wasn't found will be logged

This should be safe so long as backups that had any sort of error, including "non-catastrophic" errors are marked as failed. Marking those as failed ensures that corso is using a matched set of metadata and data in the base snapshot rather than a partial snapshot

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2550

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-17 03:35:55 +00:00
ashmrtn
7995640109
Disable kopia optimization that lets us reuse a base directory (#2522)
## Description

Just handing the base directory to kopia doesn't let us see which items have been added which means we can't properly merge backup details later on.

Future implementations could do backup details merging based on path prefixes which could fix this problem.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2521

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-16 19:57:48 +00:00
Keepers
28ad304bb7
update items to accept ctx, fault.Errors (#2493)
## Description

In order for corso to track recoverable errors,
we need to pass a fault.Errors struct into the
items stream.  As long as we're doing that, we
might as well pass along the available ctx as well.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-16 19:09:20 +00:00
dependabot[bot]
fffa633971
⬆️ Bump github.com/aws/aws-sdk-go from 1.44.201 to 1.44.202 in /src (#2538)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.201 to 1.44.202.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.202 (2023-02-15)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/cloudtrail</code>: Updates service API
<ul>
<li>This release adds an InsufficientEncryptionPolicyException type to the StartImport endpoint</li>
</ul>
</li>
<li><code>service/elasticfilesystem</code>: Updates service documentation
<ul>
<li>Documentation update for EFS to support IAM best practices.</li>
</ul>
</li>
<li><code>service/frauddetector</code>: Updates service API, documentation, and paginators</li>
<li><code>service/glue</code>: Updates service API and documentation
<ul>
<li>Fix DirectJDBCSource not showing up in CLI code gen</li>
</ul>
</li>
<li><code>service/privatenetworks</code>: Updates service API and documentation</li>
<li><code>service/rds</code>: Updates service API, documentation, waiters, paginators, and examples
<ul>
<li>Database Activity Stream support for RDS for SQL Server.</li>
</ul>
</li>
<li><code>service/wafv2</code>: Updates service API and documentation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="6cdd617804"><code>6cdd617</code></a> Release v1.44.202 (2023-02-15) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4726">#4726</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.201...v1.44.202">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/aws/aws-sdk-go&package-manager=go_modules&previous-version=1.44.201&new-version=1.44.202)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2023-02-16 18:32:57 +00:00
Abin Simon
ce37bc039f
Revert "⬆️ Bump docker/build-push-action from 3 to 4" (#2540)
Reverts alcionai/corso#2402
https://github.com/alcionai/corso/issues/2539
2023-02-16 16:55:22 +00:00
Keepers
bb69eeae4c
some fault err explanation cleanup (#2464)
## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x] 💪 Manual
2023-02-16 15:31:27 +00:00
Niraj Tolia
ffcc248398
Correctly add CA certificates (#2542)
## Description

This ensures that the CA certificates are not from some random version of Alpine but from the version we are using as a base image.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2529 

## Test Plan

- [x] 💪 Manual
2023-02-16 07:40:00 +00:00
Abin Simon
727416c6b0
Remove redundant err check (#2541)
## Description

<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-16 07:34:58 +00:00
Abin Simon
2a66097e15
Check for err in repo connect (#2537)
## Description

Add a missing err check to repo connect. Follow up to https://github.com/alcionai/corso/pull/2502 .

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-16 05:44:32 +00:00
Abin Simon
55825afdbd
Deletion related fixes for OneDrive delta incrementals (#2518)
## Description

- parentReference for deleted items will not have path.
- GC was not responding with deleted state
- Deleted items are not streamed by kopia

Borrows some changes from https://github.com/alcionai/corso/pull/2503/files as that might be delayed with on merge. Will rebase that if this gets merged fist.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2117
* fixes https://github.com/alcionai/corso/issues/2517

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-16 04:19:50 +00:00
ashmrtn
476a931ccb
Update tests in kopia package (#2514)
## Description

Use the new suite.Suite wrappers in tester package. Also update suite.Run and some nolint directives.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2373

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-16 02:22:43 +00:00
ashmrtn
034ee0df39
Use suite wrapper in internal/common package (#2527)
## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2373

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-16 01:10:13 +00:00
ashmrtn
efe1b6292d
Extra test for some recursive delete behavior (#2478)
## Description

Folders replaced after a delete should not have their subfolders reparented to the new folder.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-15 21:14:32 +00:00