717 Commits

Author SHA1 Message Date
neha_gupta
2ad32660e8
use graph client for large data set (#3355)
<!-- PR description-->
- Use graph client instead of Resty client. This brings uniformity and will help in utilising other features like Middlewares from graph client wrapper
- Moved the upload session to graph package to use Graph http-client wrapper

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2300

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-05-16 19:22:13 +00:00
ashmrtn
f8252b4cd3
Only remove previous deltas from metadata under some conditions (#3419)
Only allow removing previous delta tokens and not
previous paths. This ensures folder deletions will
be picked up if all items are enumerated

Updates tests to match new behavior, mostly
adjusting expectations around del list when
doNotMerge is applied

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2334

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-16 18:43:31 +00:00
Abin Simon
67e055360d
Bump msgraph deps to 1.0.0 (#3345)
Bumps ms graph and all dependent libs to 1.0.0

*Still can't drop betasdk as `models.SitePageable` is still not available.*

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Closes https://github.com/alcionai/corso/pull/3327
* Closes https://github.com/alcionai/corso/pull/3326
* Closes https://github.com/alcionai/corso/pull/3324
* Closes https://github.com/alcionai/corso/pull/3323
* Closes https://github.com/alcionai/corso/pull/3322
* Closes https://github.com/alcionai/corso/pull/3321
* Closes https://github.com/alcionai/corso/pull/3320
* Closes https://github.com/alcionai/corso/issues/3354

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-16 17:44:16 +00:00
Keepers
6febc3ce5b
add namespace to fault items (#3415)
prevent fault items and skips from clobbering eachother by adding a namespace to the fault item that defines a deduplication boundary.  This allows multiple items in a single service operation to have identical ids so long as their namespace differs.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3283

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-16 17:18:51 +00:00
neha_gupta
694abfab68
refactor onedrive test data creation (#3343)
<!-- PR description-->

Currently we are duplicating the Onedrive test data creation code at two places-
- in cli package 
- in test files.
Moved the common code to connector package so that both cli and test files can use the same common code.

#### Does this PR need a docs update or release note?

- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->

- [ ] 🤖 Supportability/Tests


#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3183

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
2023-05-16 09:35:26 +00:00
Abin Simon
5a2cd41d57
Convert PageLinker to DeltaPageLinker for NonDelta pagers (#3411)
This ensures that the interface for both Delta and NonDelta requests are the same and we don't have to cast the type before getting delta url from the returned object.

Follow up to https://github.com/alcionai/corso/pull/3212#discussion_r1191511216

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-16 05:26:13 +00:00
Abin Simon
a55758020f
Ensure response body gets closed (#3409)
From https://pkg.go.dev/net/http#Client.Do

> If the returned error is nil, the Response will contain a non-nil
> Body which the user is expected to close. If the Body is not both read
> to EOF and closed, the Client's underlying RoundTripper (typically
> Transport) may not be able to re-use a persistent TCP connection to
> the server for a subsequent "keep-alive" request.

These were found using https://github.com/timakin/bodyclose. Not adding it as a linter for now as it was producing false positive(I think) on OneDrive item. The close was happening in a9918d2f78/src/internal/kopia/upload.go (L68) instead of right alongside it which might be why it got thrown off.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-16 04:46:56 +00:00
Keepers
97ed97c1c3
logic change for sharepoint permission support (#3333)
adds logical changes required for sharepoint permission handling, both backup and restore.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3135

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-15 21:03:00 +00:00
Keepers
a9918d2f78
add repoRef pii transformer (#3383)
adds a transformer func in path/elements.go to safely log a repoRef (or any other dir ref) without needing to conceal the entire string.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
2023-05-12 21:43:16 +00:00
Keepers
60f6d4a035
renaming/cleanup for sharepoint perms pt.1 (#3330)
No logic changes, just renaming and minor cleanup.

PRs to follow:
1. collect various maps in onedrive collections into a single cache.
2. logic changes and tests to produce sharepoint permissions backup/restore e2e.
3. extend permission identity-type retention.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3135

#### Test Plan

- [x]  Unit test
2023-05-12 21:23:52 +00:00
Abin Simon
729543999f
Remove all uses of CORSO_URL_LOGGING (#3406)
https://github.com/alcionai/corso/pull/3212#discussion_r1191768302

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-12 16:52:40 +00:00
Abin Simon
f8aa37b822
Add non delta pagers to exchange (#3212)
When the user's mailbox is full, we cannot make use of delta apis. This adds initial changes needed to create separate delta and non delta pagers for all of exchange.

*I would suggest looking commit wise when reviewing the PR.*

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-12 15:56:49 +00:00
Keepers
1b417af5bb
minor updates to the path.Path interface (#3384)
Swaps path.Append from a single item method to accept a variadic list of string elements.  Since 95% of all calls to path.Append were items, also adds a shorthand
AppendItem func to the interface for easy clarity. Finally, adds a Last() method to elements for getting the last element in the slice.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-12 01:29:17 +00:00
ashmrtn
674d3eec91
Switch to folder IDs in Exchange (#3373)
Store all folders from Exchange by folder ID
in kopia. Remove the old duplicate folder
name stop-gap measure as well since it's no
longer required

Update tests to check LocationPath instead
of FullPath since LocationPath still has
display name info

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3197

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-05-11 23:07:38 +00:00
Keepers
6c410c298c
consts for drives, root: (#3385)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 22:46:51 +00:00
Keepers
cf2aa9013a
refine rate limiter: per-query token consumption (#3358)
Now that the rate limiter is split by service, we can further split by per-query token consumption.  Two primary service cases exist: all exchange queries assume to cost a single token (for now).  Drive-service queries are split between permissions (5), default cost (2), and single-item or delta gets (1).

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2951

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 18:23:14 +00:00
Keepers
0202207f3e
add nil pointer guard to resp.Headers in middleware (#3393)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💚 E2E
2023-05-11 16:38:13 +00:00
ashmrtn
3be3b72d0a
Combine collection structs (#3375)
They both implement the same underlying functionality, just in slightly different ways. Combine them so there's less code duplication.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-11 03:54:49 +00:00
Keepers
e7d2aeac5d
add drive-level tombstone for deleted drives (#3381)
adds a tombstone collection for any drive that has been completely deleted (or that surfaced from a prior backup, but does not exist in the current) from the driveish account.
---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3379

#### Test Plan

- [x]  Unit test
2023-05-11 03:20:28 +00:00
Keepers
c5b388a721
add indeets test helper, implement in exchange op (#3295)
Adds a helper for building expected details entries and checking
them after a backup.  Implements the helper in the exchange backup
tests in operations/backup integration.  Will follow with a onedrive
implementation.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3240

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-11 02:28:18 +00:00
Keepers
c0725b9cf9
some quick logging and error naming updates (#3348)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3344

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 02:03:31 +00:00
Keepers
522d6d2206
fix up per-service token constraints (#3378)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 01:22:05 +00:00
Keepers
833216c8ae
normalize folders restored in testing (#3363)
Normalize all test folders to use a constant prefix: Corso_Test.  This cleans up and centralizes all per- test variations on the restore destination.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 01:02:30 +00:00
neha_gupta
6c2b78de07
Item size- sum of attachment and email body (#3291)
<!-- PR description-->

Size of emails will be - sum of - size of attachment and size of email body
In case of contacts and events, since mostly everything is data we will check the size as - total serialised bytes 

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix

#### Issue(s)
* #3152

#### Test Plan
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-10 02:34:12 +00:00
ashmrtn
b508ea3b72
Update kopia restore logic to take a (RepoRef, Collection Restore Path) pair (#3337)
Begin expanding the restore logic to take a pair of
paths, one denoting the precise location of the item
in kopia and the other denoting the "restore location"
or path the item should be placed at during restore

This PR is not expected to change system functionality
at all

This is the first of 2 PRs to setup all the logic for
this. This PR does not handle properly merging
together multiple collections that have the same
restore location but different RepoRefs due to recent
updates to the kopia wrapper restore logic

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3197

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-10 01:50:38 +00:00
Keepers
33e57c0d5a
introduce per-service rate limiter configurations (#3357)
Adds a context passdown that allows GC to define the service being queried at a high level, and the rate limiter to utilize different rate limiters based on that info.  Malformed or missing limiter config uses the default limiter.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2951

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-09 22:08:36 +00:00
Keepers
dbb3bd486d
handle large item client redirects (#3350)
adds the khttp redirectMiddleware to the http wrapper used for large item downloads, to ensure that proxy servers and other 3xx class responses appropriately follow their redirection.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3344

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-09 20:12:08 +00:00
Keepers
1dcdd28a90
disguise user's default drive as enum (#3339)
When calling drive enumeration, secretly return only the user's default drive if running in onedrive, to prevent us from loading multiple drives unintentionally.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3335

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-05 22:58:36 +00:00
ashmrtn
f5365f19c5
Return map instead of using in-out param (#3308)
Return type should really be switched to []data.BackupCollection but that's a non-trivial change to make since tests rely on being able to lookup things by folder ID.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-05 21:00:16 +00:00
Abhishek Pandey
5b9cd69e29
Fetch calendar by name on 409 ErrorFolderExist (#3318)
Handle 409, `ErrorFolderExists` error in restore path while creating destination calendar. We need to fix this for all m365 services. This PR is focused on calendar only.

Context:
`CreateCalendar()` may fail with ErrorFolderExists under certain error conditions. For e.g. consider below scenario.

1. `CreateCalendar()` does a POST to graph to create restore destination calendar but this fails with 5xx.
2. It's possible that step 1 may have left some dirty state in graph. For e.g. it's possible that the destination folder in step 1 was actually created, but 5xx was returned due to other reasons.
3. So when we reattempt POST in such a scenario, we sometimes observe ErrorFolderExists error .
4. Corso should be resilient to such errors. To fix this, when we encounter such an error, we will do a GET to fetch the restore destination folder and add it to folder cache.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
Integration tests and mock tests will be added in a follow up PR. 
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-05 19:43:07 +00:00
Keepers
ef2083bc20
do not dump 2xx bodies in retry checker (#3332)
Don't dump the response body if there was no error as
the requested data could be multiple GB in size in
some cases

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-05 19:00:09 +00:00
Keepers
3a2d0876dd
consolidate aggregation of parent-item exclude map (#3258)
introduces a new type wrapping a nested map so that aggregation of globally excluded items in driveish services don't need to manage the map updates themselves.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2340

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-04 19:32:47 +00:00
Keepers
9b21699b6b
reset body reader on corso retry handling (#3280)
The kiota compressor middleware will attempt to compress the request body.  In the event that we have a corso-middleware- retriable response (eg: status 500), we need to reset the seek position of the req.Body, similar to how graph api does in their retry handler, or else the re-run of the compressor will already have read the full req.Body and the retried call will have a zero len body.

---

#### Does this PR need a docs update or release note?

- [y]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
2023-05-04 18:31:12 +00:00
ashmrtn
07b5acb92d
Temporary solution to duplicate Exchange folders (#3300)
For Exchange email or contacts backup, pick on the folder with
the highest ID in the hopes that it will be the most recently
used one

A better solution for this situation should be coming soon

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3197

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-04 01:51:53 +00:00
Keepers
6cc779eb1a
additional middleware segfault protection (#3296)
gotta catch 'em all

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3285

#### Test Plan

- [x] 💚 E2E
2023-05-03 22:10:27 +00:00
Keepers
7326730e0d
move onedrive.MetadataItem to metadata.Item (#3235)
In keeping with other changes that migrate shared metadata to the onedrive/metadata pkg for exported access.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3135

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-03 20:27:14 +00:00
Keepers
d067a79483
move permissions into metadata (#3216)
Moves the UserPermissions struct into onedrive/metadata as
the Permission struct, for better shared support across onedrive,
sharepoint, and any other packages that reference it.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3135

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-03 18:52:25 +00:00
Keepers
b2054a630d
protect retryRequset from segfaults (#3292)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3285

#### Test Plan

- [x] 💚 E2E
2023-05-03 17:33:55 +00:00
Abin Simon
d478725a04
Small typo fix in a log message (#3290)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-03 12:20:21 +00:00
Keepers
c152ce8b25
move and rename common/time.go (#3215)
moves common/time.go to common/dttm/dttm.go, and renames many of the consts and funcs with names that better explain their design and intent of usage.

Only code movement, no logical changes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-02 19:39:29 +00:00
Keepers
e72fa49018
retry on gateway timeout (#3271)
We don't currently retry on graph API gateway timeout/bad gateway responses.  This change adds those statuses to our retryable codes.

Also adds some qol clues tracking and error formatting.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-02 17:47:45 +00:00
ashmrtn
dcb9d81f3f
Fix progress folder display for Exchange (#3219)
---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3013

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-29 00:19:54 +00:00
ashmrtn
619bcd6eb8
Check for new and older content URLs (#3263)
Some requests appear to be returning an older version of the content URL key. This was causing corso backup failures. This commit expands the logic to check for both the new and old keys for the URL.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3262 

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-28 23:06:10 +00:00
Keepers
1cd592d216
minor errs cleanup (#3234)
some code cleanup that happened while investigating a bug repro

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-27 16:57:38 +00:00
ashmrtn
b8e868c05b
Return SharePoint exclude information (#3242)
Allows proper deletion of files and also fixes file moves.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3240

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-27 16:25:12 +00:00
Keepers
f2f010b9f6
cache created folders when restoring drive items (#3220)
drive items aren't currently getting cached after creation.  Instead, we do a lookup on the folder name every time we walk the folder hierarchy toward the next child to restore.  Normally this would work fine, but apparently if the folder is named `folder`, then graph api returns nothing, causing the process to fail when repeatedly creating the folder.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-26 20:53:26 +00:00
Vaibhav Kamra
6395dcbe39
Add parent path information to malware detected items (#3221)
This allows identifying the file location when an item is detected with malware

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #3112 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-26 17:17:52 +00:00
Keepers
edef23bfc4
errant cleanups before incremental support (#3201)
a handful of errant cleanups to help the PR for adding incremental support to sharepoint.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3136

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-25 21:09:41 +00:00
Keepers
0ae1a04097
use nanosecond folders for testing (#3205)
we've been failing on folder name collision, particularly in this package.  Probably because it's using second-granularity folder names.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-25 20:33:39 +00:00
Keepers
c95a07660a
ditch userInfo middle struct (#3206)
we don't need a m365.UserInfo when we already have an api.UserInfo

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-25 19:05:07 +00:00