1063 Commits

Author SHA1 Message Date
Danny
ef7d37e246
GC: Backup: Kiota Serialization Patch no longer required. (#1255)
## Description
Kiota serialization patch found located `exchange_data_collection.go`L 316 required a large collection of functions to make the message object serializable. Kiota upgrades since instantiation no longer require this collection of functions. 

Dead code removed from m365Transform.go

<!-- Insert PR description-->

## Type of change
- [x] 🗺️ Documentation


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

- [x]  Unit test
2022-12-02 21:39:44 +00:00
ashmrtn
a8d855a9da
Manual iteration for calendar events (#1628)
## Description

Will fit better with how incremental backups are planned to work in the future as next links can be extracted.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

part of:
* closes #1612 

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 20:44:07 +00:00
Danny
e767bb0b77
Backup: SharePoint: List content retrieval (#1633)
## Description
Logic for List retrieval from M365 back store. Keeping separate from Collection creation logic as there are a lot of calls to retrieve all supporting relationship data from the backstore.  
Adds collection to 
## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #1475<issue>

## Test Plan

- [x]  Unit test
2022-12-02 19:31:20 +00:00
ashmrtn
6dcbc8f2a1
Set file mod time in KopiaWrapper (#1405)
## Description

Set the mod time of uploaded files to either the mod time of the item (if it has one) or the current time (if it does not have one). Also add tests to check caching in kopia works properly.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #621 

part of:
* #547 

merge after:
* #1427 
* #1430

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 16:49:27 +00:00
ashmrtn
9963e50b99
More consistent folder cleanup for OneDrive test (#1663)
## Description

Cleanup all folders instead of just some of them. Log errors instead of failing test

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #1661

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 16:37:01 +00:00
ashmrtn
743c614ce4
Use previous snapshots to allow kopia to find previous versions of files (#1430)
## Description

Tell kopia about previous snapshots during the backup
so it can make use of them to skip uploading some data.

Currently only selects the most recent completed snapshot
and most recent incomplete snapshot that contains the
information being backed up. This can be tuned later if
it is not working good enough. However, increasing the
number of previous snapshots passed in may increase memory
usage during backup

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1404

merge after:
* #1427 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 16:25:16 +00:00
dependabot[bot]
31971bf356
⬆️ Bump actions/github-script from 5 to 6 (#1652)
Bumps [actions/github-script](https://github.com/actions/github-script) from 5 to 6.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/actions/github-script/releases">actions/github-script's releases</a>.</em></p>
<blockquote>
<h2>v6.0.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Update default runtime to node16 by <a href="https://github.com/thboop"><code>@​thboop</code></a> in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/235">actions/github-script#235</a></li>
<li>Update <code>node-fetch</code> by <a href="https://github.com/joshmgross"><code>@​joshmgross</code></a> in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/237">actions/github-script#237</a></li>
<li>Update <code>@actions/core</code> to 1.6.0 by <a href="https://github.com/joshmgross"><code>@​joshmgross</code></a> in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/238">actions/github-script#238</a></li>
</ul>
<h3>Breaking Changes</h3>
<p>With the update to Node 16 in <a href="https://github-redirect.dependabot.com/actions/github-script/issues/235">#235</a>, all scripts will now be run with Node 16 rather than Node 12.</p>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/thboop"><code>@​thboop</code></a> made their first contribution in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/235">actions/github-script#235</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a href="https://github.com/actions/github-script/compare/v5...v6.0.0">https://github.com/actions/github-script/compare/v5...v6.0.0</a></p>
<h2>v5.1.1</h2>
<h2>What's Changed</h2>
<ul>
<li>Bump <code>@​actions/core</code> to 1.9.1 by <a href="https://github.com/cory-miller"><code>@​cory-miller</code></a> in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/281">actions/github-script#281</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a href="https://github.com/actions/github-script/compare/v5.1.0...v5.1.1">https://github.com/actions/github-script/compare/v5.1.0...v5.1.1</a></p>
<h2>v5.1.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Update to latest versions for dev dependencies by <a href="https://github.com/joshmgross"><code>@​joshmgross</code></a> in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/204">actions/github-script#204</a></li>
<li>update plugin dependencies by <a href="https://github.com/PeterNitscheMI"><code>@​PeterNitscheMI</code></a> in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/216">actions/github-script#216</a></li>
<li>Update licenses and use <code>jonabc/setup-licensed</code> in workflow by <a href="https://github.com/joshmgross"><code>@​joshmgross</code></a> in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/228">actions/github-script#228</a></li>
</ul>
<h2>New Contributors</h2>
<ul>
<li><a href="https://github.com/johan-lindqvist"><code>@​johan-lindqvist</code></a> made their first contribution in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/201">actions/github-script#201</a></li>
<li><a href="https://github.com/ansgarm"><code>@​ansgarm</code></a> made their first contribution in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/215">actions/github-script#215</a></li>
<li><a href="https://github.com/PeterNitscheMI"><code>@​PeterNitscheMI</code></a> made their first contribution in <a href="https://github-redirect.dependabot.com/actions/github-script/pull/216">actions/github-script#216</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a href="https://github.com/actions/github-script/compare/v5.0.0...v5.1.0">https://github.com/actions/github-script/compare/v5.0.0...v5.1.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="d556feaca3"><code>d556fea</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/github-script/issues/300">#300</a> from actions/joshmgross/v6.3.3</li>
<li><a href="01fde8b524"><code>01fde8b</code></a> Update version to 6.3.3</li>
<li><a href="633e9fd3a1"><code>633e9fd</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/github-script/issues/279">#279</a> from nineinchnick/update-glob</li>
<li><a href="ee124b1288"><code>ee124b1</code></a> Update dist</li>
<li><a href="ca24d5fb29"><code>ca24d5f</code></a> Update <code>@actions/glob</code> license version</li>
<li><a href="c09747ec1a"><code>c09747e</code></a> Merge branch 'main' into update-glob</li>
<li><a href="100527700e"><code>1005277</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/github-script/issues/299">#299</a> from actions/joshmgross/v6.3.2</li>
<li><a href="085a7754e8"><code>085a775</code></a> Bump version to 6.3.2</li>
<li><a href="6871f0ffce"><code>6871f0f</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/actions/github-script/issues/295">#295</a> from rentziass/rentziass/update-actions-core</li>
<li><a href="7ed718295b"><code>7ed7182</code></a> Update <code>@​actions/core</code> to 1.10.0</li>
<li>Additional commits viewable in <a href="https://github.com/actions/github-script/compare/v5...v6">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=actions/github-script&package-manager=github_actions&previous-version=5&new-version=6)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2022-12-02 15:05:57 +00:00
Abin Simon
d11183658a
Show backup size for list commands (#1648)
## Description

Add size and stored size to cli output for `backup list`

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/1622

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 05:58:23 +00:00
ashmrtn
3eae0f969d
Contacts delta endpoint fetch (#1644)
## Description

Use delta endpoint to get contacts in each folder

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

part of:
* #1612 

merge after:
* #1620 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 05:30:14 +00:00
Keepers
c04b3ac6cd
adds a flag for more readable console logs (#1642)
## Description

I was getting frustrated by the readability of logs
in the console, and wanted reduce some of my
eye strain during development.
Adds a hidden "readable-logs" flag for ease of
reading logs during development.

## Type of change

- [x] 🐹 Trivial/Minor

Logs without flag:
<img width="586" alt="image" src="https://user-images.githubusercontent.com/104464746/204873851-acd368d1-2c9c-43ef-bd4b-535896e56ef0.png">

Logs with flag:
<img width="406" alt="image" src="https://user-images.githubusercontent.com/104464746/204873940-8ab3c6fd-24f8-458e-8ea9-8bdd1c27f872.png">
2022-12-02 04:57:51 +00:00
Keepers
34ba06cf61
add suffix and pathSuffix filters (#1659)
## Description

Preparing for handling sharepoint webUrl
selectors, we will need suffix-matching filters.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1616

## Test Plan

- [x]  Unit test
2022-12-02 04:44:01 +00:00
ashmrtn
34832591ce
Use delta endpoint to fetch exchange mail items (#1620)
## Description

Use delta endpoint to fetch exchange mail items. Disables picking only some item attributes as query parameters are not implemented in this version of Graph SDK.

Switches to custom pagination iteration method so that next links and delta tokens can be extracted later on.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1612 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-02 03:57:15 +00:00
ashmrtn
af977f46ca
Fix OneDrive and Kopia Wrapper test flakes (#1662)
## Description

Minor fixes for flakey tests:

* Dedupe OneDrive folders by checking IDs. Should help remove some test flakes.
* Weaken checks on kopia snapshot manager tests as they use maps with indeterminate iteration orders

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1660 
* closes #1657 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-02 03:39:45 +00:00
Abin Simon
c9a3884289
Add check using staticcheck (#1649)
## Description

Add linting using staticcheck. This is added as a separate check from golangci-lint as we were running into memory issues when running it via golangci-lint.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/pull/1634

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-02 03:28:11 +00:00
Keepers
b52a3323c0
add shell to onedrive purge (#1666)
## Type of change

- [x] 🐛 Bugfix
2022-12-02 01:09:04 +00:00
Keepers
412b73bf74
purge onedrive folders on gh action (#1655)
## Description

runs the golang purge script for onedrive folders
when running the pwsh purge scripts.  This will
move test-generated folders in onedrive into
a recycle bin.  However, that recycle bin cannot
be purged via api, and must be manually emptied
periodically for each user.

## Type of change

- [x] 💻 CI/Deployment

## Issue(s)

* #1525

## Test Plan

- [x] 💚 E2E
2022-12-01 19:38:50 +00:00
ashmrtn
e9afbaf0ec
Find previous kopia snapshots for given set of tags (#1427)
## Description

Helper functions that allow finding the most recent complete
(and maybe incomplete) snapshots for a given set of tags. The
returned snapshots can later be used to do more efficient kopia
snapshots by allowing kopia to determine it's already uploaded
a file and skip uploading it again.

The number of most recent snapshots to return can be tuned (right
now it returns 1 most recent), but may cause more memory usage
during backups.

Kopia currently has some oddities when getting snapshot
manifests via tags:
* tag values cannot be empty or the comparison returns always
  true, selecting all snapshot manifests
* kopia does not currently tag snapshot manifests made during
  checkpoints. A patch to upstream kopia is needed to fix this.
  All other manifests that match the tags will be returned though

Added code is not currently connected to any backup logic.

Viewing by commit may help as there was a tad of lift'n'shift code
movement for the sake of organization

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1404 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-01 19:07:44 +00:00
dependabot[bot]
2687ecf961
⬆️ Bump go.uber.org/zap from 1.23.0 to 1.24.0 in /src (#1647)
Bumps [go.uber.org/zap](https://github.com/uber-go/zap) from 1.23.0 to 1.24.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/uber-go/zap/releases">go.uber.org/zap's releases</a>.</em></p>
<blockquote>
<h2>v1.24.0</h2>
<p>Enhancements:</p>
<ul>
<li><a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1148">#1148</a>[]: Add <code>Level</code> to both <code>Logger</code> and <code>SugaredLogger</code> that reports the
current minimum enabled log level.</li>
<li><a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1185">#1185</a>[]: <code>SugaredLogger</code> turns errors to zap.Error automatically.</li>
</ul>
<p>Thanks to <a href="https://github.com/Abirdcfly"><code>@​Abirdcfly</code></a>, <a href="https://github.com/craigpastro"><code>@​craigpastro</code></a>, <a href="https://github.com/nnnkkk7"><code>@​nnnkkk7</code></a>, and <a href="https://github.com/sashamelentyev"><code>@​sashamelentyev</code></a> for their
contributions to this release.</p>
<p><a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1148">#1148</a>: <a href="https://github-redirect.dependabot.com/uber-go/zap/pull/1148">uber-go/zap#1148</a>
<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1185">#1185</a>: <a href="https://github-redirect.dependabot.com/uber-go/zap/pull/1185">uber-go/zap#1185</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/uber-go/zap/blob/master/CHANGELOG.md">go.uber.org/zap's changelog</a>.</em></p>
<blockquote>
<h2>1.24.0 (30 Nov 2022)</h2>
<p>Enhancements:</p>
<ul>
<li><a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1148">#1148</a>[]: Add <code>Level</code> to both <code>Logger</code> and <code>SugaredLogger</code> that reports the
current minimum enabled log level.</li>
<li><a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1185">#1185</a>[]: <code>SugaredLogger</code> turns errors to zap.Error automatically.</li>
</ul>
<p>Thanks to <a href="https://github.com/Abirdcfly"><code>@​Abirdcfly</code></a>, <a href="https://github.com/craigpastro"><code>@​craigpastro</code></a>, <a href="https://github.com/nnnkkk7"><code>@​nnnkkk7</code></a>, and <a href="https://github.com/sashamelentyev"><code>@​sashamelentyev</code></a> for their
contributions to this release.</p>
<p><a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1148">#1148</a>: <a href="https://github.coml/uber-go/zap/pull/1148">https://github.coml/uber-go/zap/pull/1148</a>
<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1185">#1185</a>: <a href="https://github.coml/uber-go/zap/pull/1185">https://github.coml/uber-go/zap/pull/1185</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="a55bdc32f5"><code>a55bdc3</code></a> Release v1.24.0 (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1208">#1208</a>)</li>
<li><a href="ed5598a9e2"><code>ed5598a</code></a> Use ReplaceAll instead of Replace with -1 pos arg (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1195">#1195</a>)</li>
<li><a href="da406e3622"><code>da406e3</code></a> Chore: Fix deprecation docs and links (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1193">#1193</a>)</li>
<li><a href="9b86a50a3e"><code>9b86a50</code></a> SugaredLogger: Turn error into zap.Error  (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1185">#1185</a>)</li>
<li><a href="9137e0e842"><code>9137e0e</code></a> go.mod: Bump go directive to 1.19 (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1181">#1181</a>)</li>
<li><a href="7cabba7950"><code>7cabba7</code></a> Open absolute paths as files, limited Windows support (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1159">#1159</a>)</li>
<li><a href="7681a0abf6"><code>7681a0a</code></a> Export objectMarshalerPtr (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1171">#1171</a>)</li>
<li><a href="9abd14b74c"><code>9abd14b</code></a> chore: remove duplicate word in comments (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1169">#1169</a>)</li>
<li><a href="eae3743bc3"><code>eae3743</code></a> Logger, SugaredLogger: Add Level method (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1148">#1148</a>)</li>
<li><a href="0d6a75bccf"><code>0d6a75b</code></a> tests: assert.Error/Contains =&gt; assert.ErrorContains (<a href="https://github-redirect.dependabot.com/uber-go/zap/issues/1158">#1158</a>)</li>
<li>See full diff in <a href="https://github.com/uber-go/zap/compare/v1.23.0...v1.24.0">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=go.uber.org/zap&package-manager=go_modules&previous-version=1.23.0&new-version=1.24.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2022-12-01 18:32:56 +00:00
dependabot[bot]
3ded147000
⬆️ Bump @iconify/react from 4.0.0 to 4.0.1 in /website (#1653) 2022-12-01 17:51:23 +00:00
dependabot[bot]
015b0874ea
⬆️ Bump minimatch, recursive-readdir and serve-handler in /website (#1654) 2022-12-01 17:35:10 +00:00
Abin Simon
a5ad6e6788
Rework CI to handle combined website and docs (#1568)
## Description

This reworks CI now that we have merged docs and website into a single deployment.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Fixes https://github.com/alcionai/corso/issues/1551

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-01 16:50:04 +00:00
Abin Simon
c58240ea3d
Parallelize OneDrive queries to get URL (#1607)
## Description

This improves the initial backup speed for OneDrive. OneDrive backup was mostly slow when we had a lot of tiny files. Case where we had mostly large files was pretty much the best case scenario and we were throttled by purely how fast we can get the files from MS and how fast kopia can process and upload it. But of small files, we were slowed by the loop which was taking quite a bit of time to fetch the download urls. We have now parallelized the query for getting the download URL. Under best case scenarios, I was able to speed it up to under 20s from ~4-5m starting point. That said, MS graph api still seems to throttle us and when that happen we still go back to around ~2m for worst case scenario. I've added 3 retries as some requests were failing when we continuously making many requests.

This should also take care of the issue of url expiring mentioned in https://github.com/alcionai/corso/issues/581 as we are only prefetching a few urls ahed of time.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1595
* fixes https://github.com/alcionai/corso/issues/581

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-01 06:11:11 +00:00
Abin Simon
784db9e12e
Speed up Exchange backup by parallelizing url fetch (#1608)
## Description

From rough numbers we can speedup an account with ~3000 emails, ~1000 contacts and ~1000 events from ~18m to <3m.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/362
* https://github.com/alcionai/corso/issues/1595
* https://github.com/alcionai/corso/pull/1607

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-12-01 04:46:24 +00:00
Keepers
58e70166d4
migrate tenant id out of event repoID (#1619)
## Description

Moves the tenantID out of the repoID hash and
into its own value (also hashed) in the event log.

Unique user identification to follow.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1602

## Test Plan

- [x] 💚 E2E
2022-12-01 00:03:21 +00:00
ashmrtn
27d0980526
Split backups by (resource owner, service) at CLI layer (#1609)
## Description

Sketch of how to split backups by (resource owner, service) for exchange. Needs more support from selectors package to be complete.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1505

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 21:50:49 +00:00
Vaibhav Kamra
c4333ad531
Use latest golangci-lint version (#1634)
## Description

Use latest golangci-lint version. Also updates cache flags.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1629 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 21:17:01 +00:00
ashmrtn
bd373bbfd4
Custom function to send get requests on delta queries (#1643)
## Description

The current version of graph SDK (v0.40.0) Corso uses does not support adding query parameters to delta requests on exchange mail folder messages or contact folder contacts. This reimplements the logic from a newer version of graph SDK so that we can add those parameters. Much of the code is lifted from the updated version of the SDK, with some combining of types that have the same fields.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1612 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 20:24:31 +00:00
ashmrtn
17a8c87168
Helper to fetch backups by ID (#1635)
## Description

Helper function to fetch backup models by ID. Also renames `Backups` to `BackupsByTag`

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1505

separated from
* #1609 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-30 16:46:32 +00:00
Abin Simon
467c3c699c
Prevent Test-Suite-Fork from running on main (#1638)
## Description

As of now, it looks like we are running `Test-Suite-Fork` in `main` instead of `Test-Suite-Trusted`. This tweaks the conditions so as to restrict the `Test-Suite-Fork` to just fork PRs.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-30 05:48:13 +00:00
Keepers
2778cd567e
add sharepoint details command (#1618)
## Description

Adds the `backup details sharepoint` command
to the CLI.  E2E support is already functional.
Additional filters will be added as needs are
identified.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1614 

## Test Plan

- [x]  Unit test
2022-11-29 23:37:05 +00:00
Keepers
f1ea464ad6
add boilerplate integration tests for sharepoint (#1600)
## Type of change

- [x] 🤖 Test

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-29 20:53:42 +00:00
Keepers
ff9a174d52
add Set func to logger package (#1631)
## Description

Add a Set(ctx, logger) func to the logger package
to allow SDK users to provide their own zap
logger instance to corso.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1630
2022-11-29 20:15:08 +00:00
Keepers
eb67db2fa5
add a discrete clone method to scopes (#1626)
## Description

Adds a method to all scopes that produces a
discrete clone of the original scope.  The result is a shallow clone of the scope, thus retaining all
original values, but is made discrete by replacing the resource owner target with a specific value.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1621

## Test Plan

- [x]  Unit test
2022-11-29 18:32:52 +00:00
ashmrtn
1d977105c0
Fix bad comparison due to timezone mismatch (#1627)
## Description

Exchange stores start/end times for events in UTC time. This fixes test failures due to comparing hours extracted from time structs directly. Without the normalization to UTC, the initial time may be different.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1625 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-11-29 17:51:34 +00:00
Niraj Tolia
cc35088be1
Add a thumbnail for the Corso demo video (#1613)
## Description

Fixes a Google search index issues

## Type of change

- [x] 🐛 Bugfix
- [x] 🗺️ Documentation
2022-11-29 01:21:27 +00:00
Danny
94145ecbf6
Path: Category expanded to include SharePoint Lists (#1610)
## Description
Backup of SharePoint requires the extension of `path.CategoryType` to be able to support the creation of `data.Collections` for `SharePoint.Lists`. 
## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Test Plan

- [x]  Unit test
2022-11-28 20:12:57 +00:00
Dhruv Patel
f05d6dec43
fix: integration tests running on PRs from forks (#1504)
## Description

* Allows integration tests to run via `repository_dispatch` events when
PRs are opened from forks

Credit: https://github.com/imjohnbo/ok-to-test

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [X] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* ALC-635

## Test Plan

<!-- How will this be tested prior to merging.-->
- [X] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-28 14:42:48 -05:00
Danny
d766c30231
GC: Populate Refactor (Contacts) (#1570)
## Description

Iterator Removal from Cache Folder Resolver.
The iterator overhead contributes to the complexities of the GC backup pipeline. Issue 1486's goal is to remove errors that are populated from non-error scenarios (e.g. container no longer exists).

Stage 1: Reduce the number of error clauses in Populate() function
Stage 2: Update the M365ID retrieval process per resolver
PR introduces the `Iterate` approach used in delta functionality to populate the contact folder cache resolver. Reduces the number of calls and variable creation during the populate function. The increase in efficiency will reduce the amount of time concurrent Mailbox queries are being made (application throttling).

## Type of change
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

- [x]  Unit test
2022-11-28 14:38:03 +00:00
dependabot[bot]
b88d8a61ad
⬆️ Bump github.com/aws/aws-sdk-go from 1.44.144 to 1.44.145 in /src (#1604)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.144 to 1.44.145.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.145 (2022-11-23)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/grafana</code>: Updates service API and documentation</li>
<li><code>service/rbin</code>: Updates service API and documentation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="d9ed7d6c12"><code>d9ed7d6</code></a> Release v1.44.145 (2022-11-23) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4636">#4636</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.144...v1.44.145">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/aws/aws-sdk-go&package-manager=go_modules&previous-version=1.44.144&new-version=1.44.145)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2022-11-24 07:28:21 +00:00
Keepers
3f7860fd0f
retrieve all drives in each site (#1599)
## Description

Extends the site drive query to include a selector
for 'system' properties, which causes graph to
return an expanded set of drives including many
which are not normally visible to end users.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-23 23:42:30 +00:00
Niraj Tolia
5c420892b5
Enable trailing slashes for docs/website (#1603)
## Description

This should fix a bunch of the errors seen by Google when indexing the website.

## Type of change

- [x] 🐛 Bugfix
- [x] 🗺️ Documentation
2022-11-23 23:29:20 +00:00
Niraj Tolia
3c32136100
Prep for docs versioning (#1598)
## Description

- Format config for consistency
- Use relative paths in MD
- Use absolute paths for imports

## Type of change

- [x] 🌻 Feature
- [x] 🗺️ Documentation
2022-11-23 21:07:32 +00:00
Niraj Tolia
d2b5e773e7
Fix README links to the website (#1597)
## Description

These got broken when we combined docs and the website

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
2022-11-23 20:04:51 +00:00
Georgi Matev
108023efd2
Improve Rudderstack event logging (#1593)
## Description

* Fire of Identify event on RepoInit to identify user for analytics tools
* Flatten Payload structure for better mapping to analytics tools
* Snake case convention for event properties

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-11-23 17:01:39 +00:00
Keepers
22ff7f5d98
errant fixes running sp backup cli (#1589)
## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #1506

## Test Plan

- [x] 💪 Manual
2022-11-23 16:50:39 +00:00
dependabot[bot]
48e3855847
⬆️ Bump github.com/aws/aws-sdk-go from 1.44.143 to 1.44.144 in /src (#1592)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.143 to 1.44.144.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/aws/aws-sdk-go/releases">github.com/aws/aws-sdk-go's releases</a>.</em></p>
<blockquote>
<h1>Release v1.44.144 (2022-11-22)</h1>
<h3>Service Client Updates</h3>
<ul>
<li><code>service/appflow</code>: Updates service API and documentation</li>
<li><code>service/kinesisanalyticsv2</code>: Updates service API and documentation</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="1356aada48"><code>1356aad</code></a> Release v1.44.144 (2022-11-22) (<a href="https://github-redirect.dependabot.com/aws/aws-sdk-go/issues/4633">#4633</a>)</li>
<li>See full diff in <a href="https://github.com/aws/aws-sdk-go/compare/v1.44.143...v1.44.144">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/aws/aws-sdk-go&package-manager=go_modules&previous-version=1.44.143&new-version=1.44.144)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2022-11-23 12:28:32 +00:00
Niraj Tolia
c3ca63a98d
Rename home page sections to reflect contents (#1572)
## Description

Makes it easier to figure out where to make changes

## Type of change

- [x] 🗺️ Documentation
2022-11-22 22:21:37 +00:00
Keepers
d6522e01ae
filter out personal sites (#1554)
## Description

Adds a server-side filter that removes all
personal (aka: oneDrive) sites from the drive
retrieval.

The approach is not my preferred method.  I'd rather
have used a filter in the query parameters.  In fact, I
have a functioning query in postman:
`$filter=NOT contains(webUrl, 'sharepoint.com/personal/')`
But I can't seem to get the graph client query to build
when I add that to the query params.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1506

## Test Plan

- [x] 💚 E2E
2022-11-22 22:17:55 +00:00
Keepers
162713dc54
add flag to retain all progress bars (#1582)
## Description

Adds the --save-progress-bars flag, which
prevents the observe package from removing
completed progress bars.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1581

## Test Plan

- [x] 💪 Manual
2022-11-22 22:00:04 +00:00
Niraj Tolia
dea9d4d609
Remove empty page (#1576)
## Description

This is getting picked up by search engines because of the sitemap

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
- [x] 🗺️ Documentation
2022-11-22 21:48:02 +00:00