2226 Commits

Author SHA1 Message Date
ashmrtn
2fe0e8643e
Add collection wrappers for serialization format (#4408)
Add a few collection wrappers that help
inject/remove serialization format since
the handling of that is still split
across layers.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4328

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-29 17:41:46 +00:00
ashmrtn
05060e7d1f
Create exported readers for injecting and retrieving serialization version info (#4378)
Previously kopia wrapper has been transparently injecting
and stripping out a "serialization version" for all items
persisted in kopia. The version thus far has been
hard-coded to `1` and has been stored in big endian format
in the first 4 bytes of every kopia file

This PR is a step towards getting serialization versions
on a per-service/per-item basis. It exposes serialization
readers that inject and strip out info to other packages

This PR also slightly changes the serialization version
format. The changes are two-fold:
* the MSB is now used to represent if the item was
  deleted between the time it was discovered and when
  its data was requested
* the serialization version number is set to uint16. This
  has no impact on existing persisted data since all
  versions are currently `1` (don't use more than
  16-bits). This size can be expanded in the future if
  needed, but is mostly to enforce the idea that higher
  order bits shouldn't be used for version numbers right
  now

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4328

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-29 15:52:05 +00:00
ashmrtn
1c520db3bc
Use generic item in SharePoint lists/pages (#4367)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-29 05:17:11 +00:00
ashmrtn
a806ab59bf
Use generic unindex item struct (#4366)
Switch all metadata files (of all types) to use the generic unindexed
item type. Transitioned items include:
  * previous paths and deltas for all services
  * site list for groups
  * drive .meta files

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-29 04:15:45 +00:00
ashmrtn
5521177aee
Create generic items without Info() (#4365)
Create generic Item implementations that
don't implement the ItemInfo interface.
These implementations can be used for
things like metadata files.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-29 03:29:15 +00:00
ashmrtn
d5cdf37369
Lock updating info in lazyItem (#4401)
Don't allow concurrent reads/writes to info in
lazyItem so we can make stronger assumptions
about state

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-29 01:50:15 +00:00
Keepers
5258ef0f36
assert correct error on s3 conn bad configs e2e (#4387)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💚 E2E
2023-09-29 00:45:16 +00:00
Keepers
9e0d464854
sanity test refactor (#4370)
refactoring the sanity tests with three goals:
1. move from env vars to cli commands so that unsupported commands fail loudly.
2. set up support for groups restore and export testing.
3. introduce some code re-use throughout.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3988

#### Test Plan

- [x] 💚 E2E
2023-09-28 23:53:15 +00:00
Keepers
a5f93f7a10
correctly attach store/provider flags (#4391)
corrects the storage and provider flag positioning to attach to the child command instead of the parent. Also corrects unit tests for flags to ensure flags are preset as expected.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-28 23:20:44 +00:00
Keepers
e5647a809d
add GroupByID to services (#4396)
Adds a groupByID call to services, and
adds CallConfig to the group by id api fn.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3988

#### Test Plan

- [x] 💚 E2E
2023-09-28 22:36:38 +00:00
Keepers
c3f94fd7f7
move drive pagers to pager pattern (#4316)
Drive pager usage currently showcases strong coupling between two layers: drive collection logic processing and drive api.  This PR separates that coupling by moving the full item enumeration process into the
API, and letting the collection logic process the
results. This acs as both a simplification of complex code, and a clearer separation of ownership between the two layers.

A detrimental side effect of this change is that drive item enumeration has moved from page-streaming
(ie: each page is fully processed before moving on to the next) and onto batch processing (ie: all items are stored in memory and processed in a single pass). Acknowledging that this is an unacceptable regression, a follow-up PR will appear shortly with better handling for stream-processing enumeration from the API layer as a standard part of the pattern for all pager
implementations.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-28 21:23:26 +00:00
Abin Simon
7964364e61
Fix nightly tests for exchange, onedrive and groups (#4384)
We were trying to delete a single backupOp multiple times across different tests. Added another backupOp to for use in the new test.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-09-28 06:25:14 +00:00
ashmrtn
db9596d223
Use non-zero time as fallback (#4390)
Use a non-zero time as the fallback for the mod time of an item during enumeration. This should avoid an issue where the mod time in details differs from the mod time in kopia due to a serialization issue.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-27 23:07:59 +00:00
Keepers
3c96eb6437
quick cleanup before next step (#4347)
adds a container-of-things to reduce mostly-unused return value bloat, and updates some func names to be more appropriate to their behavior.

No logical changes, just renaming/movement.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup
2023-09-27 17:17:31 +00:00
neha_gupta
38ef3b6ef6
call repo connect in connect S3 cmd (#4383)
<!-- PR description-->

Call repo.connect() in S3 connect command. 
Was this missed as part of - https://github.com/alcionai/corso/pull/4343/files or am I misunderstanding something. 

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manua
2023-09-27 16:37:15 +00:00
dependabot[bot]
91c0709d09
⬆️ Bump github.com/puzpuzpuz/xsync/v2 from 2.5.0 to 2.5.1 in /src (#4381)
Bumps [github.com/puzpuzpuz/xsync/v2](https://github.com/puzpuzpuz/xsync) from 2.5.0 to 2.5.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/puzpuzpuz/xsync/releases">github.com/puzpuzpuz/xsync/v2's releases</a>.</em></p>
<blockquote>
<h2>v2.5.1</h2>
<ul>
<li>Speed up built-in string hash function (<a href="https://redirect.github.com/puzpuzpuz/xsync/issues/106">#106</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="c3b502018e"><code>c3b5020</code></a> Speed up built-in string hash function (<a href="https://redirect.github.com/puzpuzpuz/xsync/issues/106">#106</a>)</li>
<li>See full diff in <a href="https://github.com/puzpuzpuz/xsync/compare/v2.5.0...v2.5.1">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=github.com/puzpuzpuz/xsync/v2&package-manager=go_modules&previous-version=2.5.0&new-version=2.5.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
2023-09-27 15:44:22 +00:00
jules
f7ab320117
fix getting owner of sites (#4376)
Fixes an issue where Get Site by id is looking for the `group` details in the additional data of the site items. It should be looking in `site.drive.owner` instead.
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E

Added a new test and tested using the `10rqc2` test tenant.

```
CORSO_CI_TESTS=true go test -run TestSiteIntegrationSuite/TestSites_GetByID -v
=== RUN   TestSiteIntegrationSuite
    tester.go:44: TestSiteIntegrationSuite run at 2023-09-26T17:34:05.383766Z
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID
    tester.go:44: TestSiteIntegrationSuite/TestSites_GetByID run at 2023-09-26T17:34:05.3883Z
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,96772abb-cfbe-496d-a770-301650888b3a,7ff3cdd8-1bf4-4e5d-999d-830a276c8490
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8be0b20e-6eff-4ac8-8505-53c7030c9fb8,316e3a01-1d93-412b-9f31-8758a267bb4a
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,32c3d871-e17d-47de-abff-b5d7e3491a31,316e3a01-1d93-412b-9f31-8758a267bb4a
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,7a8d49ec-d275-4589-8fc1-a8088ed31332,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,0bd5608f-159c-488c-ac51-cd80f747bc1d,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,4aae01c8-1d30-4202-9962-9e7b64b6bc10,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,d530b5ab-0dd9-4858-a07c-9d76a8f09b83,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,c87270a2-f375-44ec-b582-09f1b7ddd842,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,055d896c-88bc-45c3-aa4b-5af71f326bcc,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,13a2ff8b-537f-428f-aaf2-f2e4b08e4bde,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,1e16d84e-9e42-406e-b628-5d4b303309ed,41e5f79b-fe25-47c0-996b-d3e77571dbb1
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,2f9743f8-4ca3-411f-b85d-32e6441eefd2,11810f05-e8bd-45e2-9728-92e434260750
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,4d799bb6-222e-46ce-a626-51051eb2f311,11810f05-e8bd-45e2-9728-92e434260750
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,b0d4432d-432b-4681-9cdb-2ab789dfe145,ce88f380-fbaa-48aa-a3a1-e04599594e95
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,f10c5f9a-1b2b-4764-8685-d08e9883888c,ce88f380-fbaa-48aa-a3a1-e04599594e95
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8df7082a-62eb-4165-97f9-ead8cf04c0d8,11810f05-e8bd-45e2-9728-92e434260750
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,053684d8-ca6c-4376-a03e-2567816bb091,9b3e9abe-6a5e-4084-8b44-ea5a356fe02c
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,ba5dbd17-6f68-4405-b1b6-e1160c0ba746,8e229f4f-e962-42ef-a087-dc54daa2e48c
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,e946c519-47f6-443f-9544-574ab9f3966f,8e229f4f-e962-42ef-a087-dc54daa2e48c
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,846d8363-c646-496b-a603-06a2a22e0196,9b3e9abe-6a5e-4084-8b44-ea5a356fe02c
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,bf300606-e52e-41d7-a860-e5927f57a46d,8e229f4f-e962-42ef-a087-dc54daa2e48c
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,bf83915f-7454-4f2a-b3c1-1932251c714f,bf2bed00-f5ec-4551-96c1-dc7069392eb8
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,038e0095-4509-4f53-b045-0c86a7971b6a,3086932f-402c-494b-9aea-bf2bb1bf8303
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,25ca6d57-b92b-4731-9b87-23050e080eac,cf53a987-f1bb-4cea-a27e-62ee282ecca7
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8b672056-0409-46f6-98bb-4a9f5626830f,cf53a987-f1bb-4cea-a27e-62ee282ecca7
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,e1d12403-62e6-44d0-9798-15fd194858f7,69ca49fc-e607-438b-ba24-3bb40fc7baba
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,4892edf5-2ebf-46be-a6e5-a40b2cbf1c1a,38ab6d06-fc82-4417-af93-22d8733c22be
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,6884eaa6-108e-42ee-ac1e-8f8d344d5e53,42c29db5-bf88-46b7-af47-05a5ca15c1b3
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,5cdb15ec-4440-4115-be94-d9221b9135b3,42c29db5-bf88-46b7-af47-05a5ca15c1b3
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,4637d353-1c6d-4967-98a3-a75db4e1a925,42c29db5-bf88-46b7-af47-05a5ca15c1b3
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8b510176-8837-4a81-a26e-2020565adb92,276af97a-8a5e-4001-aa86-53958ac9548c
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,3d374d63-f5fa-4c6a-b5c3-119280bd5442,42c29db5-bf88-46b7-af47-05a5ca15c1b3
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8fa2edf7-1f13-47e5-834d-ccdb81879302,276af97a-8a5e-4001-aa86-53958ac9548c
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,7a8dccfa-aa6b-4a75-b7bd-eab3f85af9b0,859ccc3c-2ed3-4258-84d1-ec735b39adc1
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,54bc7530-a81c-4d90-8cfb-a8625168187f,859ccc3c-2ed3-4258-84d1-ec735b39adc1
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,2d6ba44f-0638-4950-8fee-537c854e9e28,859ccc3c-2ed3-4258-84d1-ec735b39adc1
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,23283ae3-170b-4db6-b50f-895b12373524,859ccc3c-2ed3-4258-84d1-ec735b39adc1
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,f238d00a-af70-4273-9017-6042a6d8a8e1,859ccc3c-2ed3-4258-84d1-ec735b39adc1
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,ee9af367-a8ef-4f56-99ad-72f53d1cd64f,859ccc3c-2ed3-4258-84d1-ec735b39adc1
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,291e91dd-f9f9-4e0f-8abc-269ade15e1be,a470fff3-c3a7-456a-98c3-ad51d166e222
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,5edef5e8-6654-4fb7-870b-52365d9818f4,859ccc3c-2ed3-4258-84d1-ec735b39adc1
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,203db13c-aa6b-4cb4-bb11-7b5ce9c2b5b6,7f2cd296-9f7c-44e1-af81-3c06d0d43007
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,b8fff07f-350b-4f62-b35f-854b0920f180,7f2cd296-9f7c-44e1-af81-3c06d0d43007
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,26b58876-c2c2-4bee-9fae-a2acbb0ebeb0,7f2cd296-9f7c-44e1-af81-3c06d0d43007
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,b39c8064-43bf-4740-83cf-8d4814ee8591,7f2cd296-9f7c-44e1-af81-3c06d0d43007
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,01af6484-492e-4bae-bd3b-ed7d14ed3369,7f2cd296-9f7c-44e1-af81-3c06d0d43007
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,6e27eab3-0ccc-4ce1-80ae-b32912c9468d,7f2cd296-9f7c-44e1-af81-3c06d0d43007
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,bb5d5b4a-e089-4e66-9868-9e263ecc635d,4fa3a2c0-fa81-4e6f-8e8b-1479a8927bc6
=== RUN   TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,009d6803-aa00-4db7-8233-e2cbd7d9f503,5408143c-ebd0-4130-82e6-767375686853
--- PASS: TestSiteIntegrationSuite (30.98s)
    --- PASS: TestSiteIntegrationSuite/TestSites_GetByID (30.97s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,96772abb-cfbe-496d-a770-301650888b3a,7ff3cdd8-1bf4-4e5d-999d-830a276c8490 (0.61s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8be0b20e-6eff-4ac8-8505-53c7030c9fb8,316e3a01-1d93-412b-9f31-8758a267bb4a (0.64s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,32c3d871-e17d-47de-abff-b5d7e3491a31,316e3a01-1d93-412b-9f31-8758a267bb4a (0.81s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,7a8d49ec-d275-4589-8fc1-a8088ed31332,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809 (0.59s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,0bd5608f-159c-488c-ac51-cd80f747bc1d,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809 (0.61s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,4aae01c8-1d30-4202-9962-9e7b64b6bc10,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809 (0.65s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,d530b5ab-0dd9-4858-a07c-9d76a8f09b83,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809 (0.61s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,c87270a2-f375-44ec-b582-09f1b7ddd842,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809 (0.69s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,055d896c-88bc-45c3-aa4b-5af71f326bcc,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809 (0.71s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,13a2ff8b-537f-428f-aaf2-f2e4b08e4bde,cd8ae214-0f28-43e1-8ec4-ad3b9fb11809 (0.65s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,1e16d84e-9e42-406e-b628-5d4b303309ed,41e5f79b-fe25-47c0-996b-d3e77571dbb1 (0.57s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,2f9743f8-4ca3-411f-b85d-32e6441eefd2,11810f05-e8bd-45e2-9728-92e434260750 (0.62s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,4d799bb6-222e-46ce-a626-51051eb2f311,11810f05-e8bd-45e2-9728-92e434260750 (0.79s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,b0d4432d-432b-4681-9cdb-2ab789dfe145,ce88f380-fbaa-48aa-a3a1-e04599594e95 (0.52s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,f10c5f9a-1b2b-4764-8685-d08e9883888c,ce88f380-fbaa-48aa-a3a1-e04599594e95 (0.51s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8df7082a-62eb-4165-97f9-ead8cf04c0d8,11810f05-e8bd-45e2-9728-92e434260750 (0.62s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,053684d8-ca6c-4376-a03e-2567816bb091,9b3e9abe-6a5e-4084-8b44-ea5a356fe02c (0.55s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,ba5dbd17-6f68-4405-b1b6-e1160c0ba746,8e229f4f-e962-42ef-a087-dc54daa2e48c (0.63s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,e946c519-47f6-443f-9544-574ab9f3966f,8e229f4f-e962-42ef-a087-dc54daa2e48c (0.66s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,846d8363-c646-496b-a603-06a2a22e0196,9b3e9abe-6a5e-4084-8b44-ea5a356fe02c (0.50s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,bf300606-e52e-41d7-a860-e5927f57a46d,8e229f4f-e962-42ef-a087-dc54daa2e48c (0.73s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,bf83915f-7454-4f2a-b3c1-1932251c714f,bf2bed00-f5ec-4551-96c1-dc7069392eb8 (0.56s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,038e0095-4509-4f53-b045-0c86a7971b6a,3086932f-402c-494b-9aea-bf2bb1bf8303 (0.60s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,25ca6d57-b92b-4731-9b87-23050e080eac,cf53a987-f1bb-4cea-a27e-62ee282ecca7 (0.54s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8b672056-0409-46f6-98bb-4a9f5626830f,cf53a987-f1bb-4cea-a27e-62ee282ecca7 (0.52s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,e1d12403-62e6-44d0-9798-15fd194858f7,69ca49fc-e607-438b-ba24-3bb40fc7baba (0.51s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,4892edf5-2ebf-46be-a6e5-a40b2cbf1c1a,38ab6d06-fc82-4417-af93-22d8733c22be (0.49s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,6884eaa6-108e-42ee-ac1e-8f8d344d5e53,42c29db5-bf88-46b7-af47-05a5ca15c1b3 (0.53s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,5cdb15ec-4440-4115-be94-d9221b9135b3,42c29db5-bf88-46b7-af47-05a5ca15c1b3 (0.50s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,4637d353-1c6d-4967-98a3-a75db4e1a925,42c29db5-bf88-46b7-af47-05a5ca15c1b3 (0.55s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8b510176-8837-4a81-a26e-2020565adb92,276af97a-8a5e-4001-aa86-53958ac9548c (0.53s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,3d374d63-f5fa-4c6a-b5c3-119280bd5442,42c29db5-bf88-46b7-af47-05a5ca15c1b3 (0.54s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,8fa2edf7-1f13-47e5-834d-ccdb81879302,276af97a-8a5e-4001-aa86-53958ac9548c (0.53s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,7a8dccfa-aa6b-4a75-b7bd-eab3f85af9b0,859ccc3c-2ed3-4258-84d1-ec735b39adc1 (0.61s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,54bc7530-a81c-4d90-8cfb-a8625168187f,859ccc3c-2ed3-4258-84d1-ec735b39adc1 (0.54s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,2d6ba44f-0638-4950-8fee-537c854e9e28,859ccc3c-2ed3-4258-84d1-ec735b39adc1 (0.67s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,23283ae3-170b-4db6-b50f-895b12373524,859ccc3c-2ed3-4258-84d1-ec735b39adc1 (0.80s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,f238d00a-af70-4273-9017-6042a6d8a8e1,859ccc3c-2ed3-4258-84d1-ec735b39adc1 (0.74s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,ee9af367-a8ef-4f56-99ad-72f53d1cd64f,859ccc3c-2ed3-4258-84d1-ec735b39adc1 (0.76s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,291e91dd-f9f9-4e0f-8abc-269ade15e1be,a470fff3-c3a7-456a-98c3-ad51d166e222 (0.61s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,5edef5e8-6654-4fb7-870b-52365d9818f4,859ccc3c-2ed3-4258-84d1-ec735b39adc1 (0.59s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,203db13c-aa6b-4cb4-bb11-7b5ce9c2b5b6,7f2cd296-9f7c-44e1-af81-3c06d0d43007 (0.63s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,b8fff07f-350b-4f62-b35f-854b0920f180,7f2cd296-9f7c-44e1-af81-3c06d0d43007 (0.62s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,26b58876-c2c2-4bee-9fae-a2acbb0ebeb0,7f2cd296-9f7c-44e1-af81-3c06d0d43007 (0.58s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,b39c8064-43bf-4740-83cf-8d4814ee8591,7f2cd296-9f7c-44e1-af81-3c06d0d43007 (0.67s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,01af6484-492e-4bae-bd3b-ed7d14ed3369,7f2cd296-9f7c-44e1-af81-3c06d0d43007 (0.69s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,6e27eab3-0ccc-4ce1-80ae-b32912c9468d,7f2cd296-9f7c-44e1-af81-3c06d0d43007 (0.63s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,bb5d5b4a-e089-4e66-9868-9e263ecc635d,4fa3a2c0-fa81-4e6f-8e8b-1479a8927bc6 (0.67s)
        --- PASS: TestSiteIntegrationSuite/TestSites_GetByID/site_10rqc2.sharepoint.com,009d6803-aa00-4db7-8233-e2cbd7d9f503,5408143c-ebd0-4130-82e6-767375686853 (0.57s)
PASS
ok  	github.com/alcionai/corso/src/pkg/services/m365	31.635s
```
2023-09-26 23:22:14 +00:00
Keepers
a8a2aee99d
remove expand drive from root site (#4377)
missed the root site call usage of expand drive
on the last fix.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4337

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-26 22:32:15 +00:00
Keepers
f37e58ee5c
graph debug logging improvements (#4368)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests
- [x] 🧹 Tech Debt/Cleanup
2023-09-26 22:01:00 +00:00
Keepers
0474e5c957
separate delta not supported from invalid (#4375)
groups backup has a failing case where delta
queries are not supported, but we aren't distinguishing
that case from the invalid delta case as expected. This
results in backup failures, instead of success with no data.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3988

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-26 18:37:15 +00:00
ashmrtn
8791c59c17
Fix NPE in Exchange CLI restore E2E setup suite (#4374)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-26 16:57:42 +00:00
neha_gupta
adbc7fe03e
allow delete of multiple backup IDs (#4335)
<!-- PR description-->

allow deletion of multiple IDs in single command

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4119

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] 💚 E2E
2023-09-26 08:53:14 +00:00
ashmrtn
5c4419fdc0
Use generic item in Drive backup (#4357)
Use generic lazy item implementation for drive
backups. This is a bit of a stop-gap
implementation since the drive logic is more
complex than exchange logic

Refactoring the drive logic to streamline it
would help reduce the number of functions
called to download a single item etc.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-26 02:50:14 +00:00
ashmrtn
a91c823e42
Generic item api update (#4356)
Minor updates to error handling for
generic item implementation

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-26 01:59:57 +00:00
ashmrtn
6690fd3ea2
Groups generic item (#4353)
Use generic item structs for Groups

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-26 00:27:43 +00:00
Keepers
365078a861
split repo create and init/cnn (#4343)
splits the functionality for creating a new repository struct, and initializing or connecting to it.  This is the first step in a multi-step improvement for init and connect processing.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2025

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-25 23:48:53 +00:00
ashmrtn
cf3744bcfb
Exch generic item (#4351)
Use generic item structs in Exchange code

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-25 23:16:12 +00:00
ashmrtn
0df876d106
Disable Exchange kopia assisted incrementals (#4360)
Can be reenabled when we get a good
solution to deletion markers for
individual items.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2023

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-25 22:23:15 +00:00
Keepers
89860d1221
add standardized tests for flags in backup cmds (#4294)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
2023-09-25 21:28:17 +00:00
Keepers
01f61a44ed
move drive expand to config property (#4362)
the addition of the drive expand requires onedrive permissions in order to retrieve sites by ID.  This causes some users to violate customer contract by requiring unwanted permissions.  This change allows configuration of the getbyID call to optionally include the drive extend, which enables the backup process to return to the permission set as expected from v0.12

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4337

#### Test Plan

- [ ] 💪 Manual
- [ ] 💚 E2E
2023-09-25 19:18:16 +00:00
Keepers
732bac17a4
add missing recoverable errs (#4363)
groups collection item streaming is missing
recoverable errors, and is only logging at this time. This may be causing backups to succeed even
when errors should cause a failure.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3988

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-25 18:35:47 +00:00
ashmrtn
a3dabaa589
Make generic version of item structs (#4350)
Generic structs implement items that have
details associated with them. Items that
don't have details will need a separate
struct since we use a downcast to
determine if we can get details.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-25 17:21:26 +00:00
Keepers
0545365b12
move container enum to pager pattern (#4298)
moves exchange container enumeration funcs in
the api to the pager pattern established with item enumeration.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-22 17:34:06 +00:00
Keepers
dcffc61bf5
do not mark not-found items as skipped (#4342)
This was an accidental mis-use of the skipped item pattern.  Items deleted during in flight during backup due to race conditions do not count as permanent
skips in the way that other skipped items do.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #4044

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-22 16:39:16 +00:00
neha_gupta
8ce6dc4217
skip-permissions flag to stop restoring of permissions of Onedrive (#4313)
<!-- PR description-->

add `skip-permissions` flag for permission restore

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3854

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-22 06:51:00 +00:00
Abin Simon
496b725035
Do not try to restore channel messages for Groups (#4334)
<!-- PR description-->

As of now, we try to restore channel messages and fail with the following error:

```
Error: Failed to run Groups restore: running restore: restoring collections: data category not supported
```

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-22 06:10:26 +00:00
ashmrtn
738693a1d7
Create and use generic tombstone collection (#4339)
Incremental backups requires us to mark some
folders as deleted by creating a collection
with state `data.DeletedState`

This PR creates a simple, generic "tombstone"
collection that does just that

The PR additionally uses the tombstone
collection in place of more complicated
implementations where any easy switch is
possible. Deleted collections in OneDrive
require more work since tests attempt to
cast to a concrete type

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4191

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-22 01:22:53 +00:00
Keepers
6bc40ea2aa
add throttling, token counters to operation (#4318)
Adds context embedding to pkg/count, and embeds
the backup, restore, and export operation ctx with the operations count.bus.  The bus is extracted from the ctx in api middleware to track throttling response counts, as well as token consumption counts.  Counts are globally tracked, not on a per-time or per resource basis.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-22 00:43:16 +00:00
Keepers
bf3a65a351
bump clues to latest version (#4326)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests
2023-09-21 23:59:15 +00:00
ashmrtn
09aebc4a31
Wire backup cleanup into maintenance operation (#4091)
Wire backup cleanup into maintenance operation.
Currently it runs under a different mode so it
won't affect existing users of the maintenance
command. Once we're satisfied with testing we
can remove the temporary const and run this
when complete maintenance is run

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3217

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-21 22:15:29 +00:00
ashmrtn
b33231e98d
Use backupBase internally in base finder code (#4315)
First step to consolidating fields in backupBases.
This adds the consolidated fields to the code for
finding bases.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3943

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-21 19:34:59 +00:00
ashmrtn
66a1a3a39d
Add helper function to get category (#4325)
Expand BaseCollection to have a helper function that returns the Category. Attempts to source from the FullPath and falls back to the PreviousPath if necessary.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4319

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-21 16:47:48 +00:00
Abin Simon
ccf8df3548
Bump msgraph sdk to 1.19.0 (#4314)
Needed some changes in the func names.

``` bash
$ sed -i "s/By\([a-zA-Z]*\)IdString/By\1Id/g" **/*.go
```

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-09-21 07:01:07 +00:00
Abhishek Pandey
7050b0c9d2
Improve error handling during backup persistence (#4320)
<!-- PR description-->

We were not catching missing snapshot ID sooner. Fixed it.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4305

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-21 05:52:48 +00:00
Abin Simon
a28f57a6de
Use HumanString for export folder names (#4276)
`Libraries` is what gets used for SP libraries and using `channelMessages` here did not look good.

Ref: fe77c30e84/src/internal/m365/service/sharepoint/export.go (L49-L52)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-21 05:05:11 +00:00
ashmrtn
231c0a65a0
Use BaseCollection in groups implementation (#4324)
Use common implementation for boiler plate
functionality

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4319

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-21 01:04:04 +00:00
Keepers
6159668b1d
utilize format validation check (#4288)
adds usage of the export format validation check
to export cli commands.  Also moves the restore
cfg validation check out of the service flags
validation checks and into the generic restore runner for better separation of concerns.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3988 

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-21 00:24:12 +00:00
ashmrtn
c3e63413d4
Fix extracting category from BaseCollection (#4323)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-09-20 22:44:15 +00:00
ashmrtn
f767b67eca
Move BaseCollection to different package (#4322)
Prep for use in other packages. Removed
tests were covered by other code

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #4319

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-09-20 22:06:33 +00:00
Keepers
c1ec3c6648
formalizes ErrApplicationThrottled (#4317)
adds graph error identification for application throttled and adds an ApplicationThrottled sentinel to pkg/errs for sdk consumer identification.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-20 21:27:01 +00:00