129 Commits

Author SHA1 Message Date
Keepers
b331f38654
fixup mailbox info (#3189)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-04-24 20:06:41 +00:00
Abhishek Pandey
41f742eba2
Add concurrency limiter middleware package (#3182)
<!-- PR description-->


Graph service only allows [4 concurrent requests per exchange
mailbox.](https://learn.microsoft.com/en-us/graph/throttling-limits#outlook-service-limits)
We are currently not honoring this limit in corso. This causes 429
errors with “Application is over its MailboxConcurrency limit”. This PR
introduces a concurrency limiter middleware. This middleware is
selectively added for exchange backups only.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E

---------

Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
2023-04-24 11:16:20 -07:00
Keepers
62daf10213
migrate onedrive using prefix collection (#3122)
#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-24 17:36:10 +00:00
Keepers
6405c8246e
move parallelism logs/checks upstream (#3003)
The fetch paralellism checks and logs occur on
every item streamed from GC.  This is a bit chatty,
and has been moved upstream in the process for
a more centralized behavior.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-04-21 20:23:41 +00:00
Keepers
d5fac8a480
custom http wrapper for item downloads (#3132)
the http client customized by the graph client for use in downloading files doesn't include our middleware like expected. This introduces a new http client wrapper that populates the roundtripper with a middleware wrapper that can utilize our kiota middleware in the same way as the graph client does.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3129

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-20 23:20:23 +00:00
neha_gupta
de1ae9f30a
Shared mailbox (#3106)
<!-- PR description-->

Handle user's API to provide user purpose of users

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-20 18:01:12 +00:00
Vaibhav Kamra
2050a953fe
Handle SyncStateInvalid error when ImmutableIDs are enabled after initial backup (#3176)
Handles the `SyncStateInvalid` error if a delta URL constructed without the immutable ID header is used.

This will fallback to full backup similar to how we handle other sync errors for delta queries.

Also adds a hidden flag so we can test this using the CLI.

*Note*: Enabling immutable IDs for exchange is not yet supported

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-19 23:52:08 +00:00
Keepers
09ef350bc0
retry on "connection reset by peer" (#3138)
retry when we receive a "connection reset by peer" error

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3129

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-04-19 18:51:29 +00:00
ashmrtn
9664afaa12
Restore with OneDrive/SharePoint ID (#3151)
Allow restoring OneDrive and SharePoint items by
M365 ID as well as file name and ShortRef

Does create an edgecase where attempting to
restore an item named the same as another item's
M365 ID causes both items to be restored

Moves the consts for OneDrive file suffixes to a
different package so that the selectors package
can also import them without a cycle

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3142

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-19 15:30:03 +00:00
Keepers
b0869fad4d
add sensitive info flag to cli (#3000)
The sensitive-info flag allows CLI users to specify the Concealer algorithm used to handle PII and
other sensitive info in logging.  Default alg is 'hash', which produces a truncated hmacsha256 string.
'mask' for flat '***' replacement and 'plaintext'
are also available.

SDK consumer are expected to configure clues
themselves, rather than specifying a config option.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2024

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-04-14 17:07:14 +00:00
Keepers
482b690e06
fix overlapping bases after fallback union (#3118)
In a case where the current manifests and the
fallback contain both distinct and overlapping
categories, the end result contains multiple
bases for the same category.  Verifydistinctresons
didn't catch this because it includes the resource owner.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
2023-04-14 16:29:29 +00:00
Abin Simon
6c64eb0c16
Fetch individual attachments in case of full download failure (#3086)
Failure can happen as the list of attachment and they data can get
pretty big.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-13 05:51:58 +00:00
Abhishek Pandey
ec064e539b
Add misc typo fixes (#3006)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?


- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->

- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
2023-04-12 23:30:38 +00:00
Keepers
da8ac5cdbc
moving code around, no logic changes (#3025)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
2023-04-12 17:16:05 +00:00
Keepers
30c86797d3
always include delta metadata, additional logging (#3092)
Ensure exchange always includes the delta metadata entry, even if it has zero deltaURLs to record.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* fixes #3019

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-12 00:43:55 +00:00
ashmrtn
27909592b8
Exit early if context is marked as done (#2883)
Keeps the code from blocking and then failing immediately without sending a retry.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2882

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-10 18:38:14 +00:00
Keepers
896c05f623
look up users and sites by id or name (#2973)
Adds a lookup step to graph connector to find
an owner's id and name given some identifier.
The identifier, for either sites or users, can be a
well formed id or name.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-07 22:29:09 +00:00
Keepers
3cc29649ed
Add GetUserInfo to services/m365 (#2991)
Adds an interface for GetUserInfo, which gets
corso-specific metadata for a single user.  Also
does some refactoring around discovery for
better interface consistency, both as variables
and as access layers.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-07 00:36:50 +00:00
Abin Simon
baddd9fc83
Get OneDrive item with getM365 (#2791)
Add support for onedrive in getM365

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-06 05:32:03 +00:00
Keepers
74e4a094e0
conceal url pii (#3015)
Adds a new package to common (pii) with a clues.
Concealer compliant struct that can log and report
url values with a mix of hidden pii and exposed
public values.  Exposed values are based on
well-known graph api path and query params.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
2023-04-04 21:01:37 +00:00
Keepers
fbb867a08d
add clues.Hide to various logs (#2994)
Begins hiding PII with clues.Hide injections in our structured logging aggregators.  This is a Non-
comprehensive change, and improvements here will be ongoing.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-04-04 16:59:31 +00:00
Abin Simon
4cc0397075
Log responses for 503 (#3012)
Currently we were logging rate limiting values for 503 which is not available and end up with empty values. Switching it to logging the response, similar to 404 as that can be more useful.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-03 17:32:44 +00:00
ashmrtn
91e0a27831
Populate location in Exchange container resolver (#2828)
Always generate both a path of IDs and a path of
display names for folders in the container resolver.

Changes in internal/connector/exchange/service_functions.go
keep the behavior of the overall system from changing

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

merge after:
* #2808

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-29 01:19:59 +00:00
Keepers
940892fc07
correct input aliasing (#2960)
Updates and corrects input aliasing according to
the following rules (in priority order):
1. if the library name is usable, use it
2. if not, alias to the package name
3. if the package name is weird, alias sensibly
4. in case of collision, alias more distant imports
5. aliases should be consistent throughout

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
2023-03-28 20:48:22 +00:00
Keepers
214206b75e
avoid segfaults in metrics middleware (#2972)
#### Type of change

- [x] 🐛 Bugfix
2023-03-28 20:13:58 +00:00
ashmrtn
79b6c0957f
Re-enable import order linting (#2928)
Fixes current errors as well

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2927

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-28 16:45:28 +00:00
Keepers
9f0e259764
metrics PoC for github.com/armon/go-metrics (#2884)
Adds a logger-oriented metrics tracker to aggregate statistics like requests-per-minute and average
http request time.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2212

#### Test Plan

- [x] 💪 Manual
2023-03-28 15:57:55 +00:00
Keepers
9d73d0c8e2
replace errors.* with clues.* (#2924)
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.

Might follow up with a linter to enforce this change.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-28 04:01:54 +00:00
Keepers
328d75945b
PoC for centralized throttling control (#2887)
PoC for using a centralized middleware to control
1k-per-minute get requests to graph api.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💪 Manual
2023-03-24 00:40:08 +00:00
Keepers
b82c994b91
Always make empty collections for prefix directories (#2929)
Use empty collections to ensure the prefix directories are available in
kopia no matter what user data we find.

This helps ensure we have some set of things we can assume about
non-failed backups including:

    They have a valid snapshot ID for data
    They have a valid snapshot ID for backup details

Eventually, this will allow us to say that backups using a base that
doesn't have a prefix directory is invalid in some way

SharePoint tests are disabled because lists and pages ignore the none
target in selectors

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #2550

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-24 00:04:03 +00:00
Abin Simon
8bebf579a4
Remove global 10 day timeout (#2875)
This has been fixed upstream with https://github.com/microsoft/kiota-http-go/pull/71 . We don't need this hack anymore.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2849
* https://github.com/alcionai/corso/pull/2694
* closes https://github.com/alcionai/corso/pull/2893
* closes https://github.com/alcionai/corso/pull/2892
* https://github.com/alcionai/corso/pull/2891

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-22 13:37:30 +00:00
ashmrtn
1aa315412a
Add more logging to middleware and incremental backup operation logic (#2899)
Ensures:
* context clues are added to logged messages
* kv-pairs added with `log.With` are added to logged messages
* add log messages for how many details items were sourced from a base snapshot during merging
* ensure log messages for base selection have all the info they should

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-21 22:33:19 +00:00
Vaibhav Kamra
c49e4bbef4
Handle 503 retries where the Retry-After header exceeds what the Graph SDK will wait for (#2888)
This logic will only trigger if the Graph retry handler does not handle the 503 (it is invoked first).

We remove the check in Corso for `absoluteMaxDelay` because our retry config will not trigger that limit (180s).
This allows the retry to kick in if the 503 response contains a `Retry-After` value > 180s

This could help with issues like https://github.com/alcionai/corso/issues/2876

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2876 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-21 05:37:24 +00:00
ashmrtn
60d173b25b
Log retry after for throttled requests (#2847)
This seems to be the header that is used to communicate the delay in seconds to wait.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2846

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-18 00:14:03 +00:00
Keepers
621e850531
add 10 day deadline to cli context (#2852)
Ensures all CLI usage has a context deadline so that we
don't bump into the graph api 100 second deadline.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
2023-03-17 23:48:13 +00:00
Keepers
91328d9898
cleanup graph logging, remove 2xx test logs (#2836)
Removes 2xx query logging from CI testing, and
slightly refactors the logging middleware to make
better use of clues and minimize boilerplate.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Test
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💚 E2E
2023-03-17 16:00:22 +00:00
Keepers
4515d3a4b2
remove unused test scoping envs (#2648)
Doesn't seem like we're using these

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Test
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2373

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-16 08:00:49 +00:00
Keepers
76b3fe3b86
append ToCore to all errors tests (#2793)
In order to retrieve all clues structured error data in tests, we need to extract it from the error using the clues library.

This change appends `clues.ToCore(err)` to all
variations of `assert.NoError(t, err)`.  The only
other changes are those necessary to preserve
linting, or to produce an error variable for the
ToCore call.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Test
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-15 19:02:47 +00:00
Abin Simon
0125876192
Skip items returning 404 (#2805)
Treat a 404 as an unavailable item and skip it similar to malware items.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
Treat a 404 as an unavailable item and skip it similar to malware items.
2023-03-15 17:52:22 +00:00
Keepers
dc21f4ce07
add a QoL path.Build func (#2763)
Adds a path.Build() func that can arbitrarily build a path according to expected standards without first initializing a path.Builder{}.
---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-13 16:56:31 +00:00
Keepers
871f0b406d
introduce fault items (#2710)
Adds the item struct to the fault package for tracking serializable and dedupliatable error sources.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🌻 Feature

#### Issue(s)

* #2708

#### Test Plan

- [x]  Unit test
2023-03-10 17:23:19 +00:00
Keepers
f4c6e65fa3
remove common errors (#2597)
## Description

Now that clues is in place, we don't need the
errors.As handling provided by common errors.
Current errors.As can be replaced with either
errors.Is checks, or HasLabel checks on the
status code.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-10 00:55:33 +00:00
Keepers
21a5729947
backup malware catch (#2703)
assuming it's possible for a graph
item to skip the malware detection,
we still want to catch and handle
400's from attempted malware
downloads downstream.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🌻 Feature

#### Issue(s)

* #2701

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-03-08 18:13:30 +00:00
Vaibhav Kamra
9c1d04e3f1
Remove Graph call that could cause throttling issues (#2727)
The logic added in [2632](https://github.com/alcionai/corso/pull/2632) introduces an extra Graph call for
every item that is being backed up - which can lead to throttling errors.

Removing this till we figure out the right solution.

This commit also adds some extra logging to help debug throttling errors.

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2632 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-08 04:05:29 +00:00
Keepers
1ca49c53a9
remove wrap and append support (#2589)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-08 01:24:48 +00:00
Danny
030147ddcb
Corso: Exchange: Mail: Backup Details to include recipients --JSON ONLY (#2689)
<!-- Insert PR description-->
#### Adds  Email recipients to the backup details for Exchange Mail. If the recipients are of length greater than one, the list of individuals is included and separated by a `,`

---

#### Does this PR need a docs update or release note?
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes  #2687<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x]  Unit test
2023-03-07 02:32:38 +00:00
Abin Simon
74840f035d
Graph ResyncRequired error can also manifest as resyncRequired (#2688)
The only change is that the first letter is now small.

<!-- Insert PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-06 05:26:43 +00:00
ashmrtn
83fe83a367
Exclude metadata files from the set that we merge during incrementals (#2650)
Metadata files don't need to appear in backup
details, which means they also shouldn't be
merged during incremental backups. If we
expect them to be merged then we'll fail the
item count check for the merge process

This excludes items with metadata extensions
from merging by not adding the to the list
of items to merge

Also make it so that metadata items don't
create backup details entries in the first
place

Manually tested to ensure
restore(backup(onedrive)) still works as
expected
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2498

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-02 13:49:19 +00:00
Keepers
ebf597b31d
logging tweaks (#2663)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🐛 Bugfix

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
2023-02-27 21:29:20 +00:00
Keepers
29c6ac4874
remove errors from connector status (#2586)
## Description

Now that fault is in place, we can remove the error tracking functionality of graph status, and let that focus purely on metrics.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-27 16:50:12 +00:00