318 Commits

Author SHA1 Message Date
Abin Simon
f8aa37b822
Add non delta pagers to exchange (#3212)
When the user's mailbox is full, we cannot make use of delta apis. This adds initial changes needed to create separate delta and non delta pagers for all of exchange.

*I would suggest looking commit wise when reviewing the PR.*

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-12 15:56:49 +00:00
Keepers
1b417af5bb
minor updates to the path.Path interface (#3384)
Swaps path.Append from a single item method to accept a variadic list of string elements.  Since 95% of all calls to path.Append were items, also adds a shorthand
AppendItem func to the interface for easy clarity. Finally, adds a Last() method to elements for getting the last element in the slice.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-12 01:29:17 +00:00
ashmrtn
674d3eec91
Switch to folder IDs in Exchange (#3373)
Store all folders from Exchange by folder ID
in kopia. Remove the old duplicate folder
name stop-gap measure as well since it's no
longer required

Update tests to check LocationPath instead
of FullPath since LocationPath still has
display name info

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3197

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-05-11 23:07:38 +00:00
Keepers
c5b388a721
add indeets test helper, implement in exchange op (#3295)
Adds a helper for building expected details entries and checking
them after a backup.  Implements the helper in the exchange backup
tests in operations/backup integration.  Will follow with a onedrive
implementation.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3240

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-11 02:28:18 +00:00
Keepers
833216c8ae
normalize folders restored in testing (#3363)
Normalize all test folders to use a constant prefix: Corso_Test.  This cleans up and centralizes all per- test variations on the restore destination.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 01:02:30 +00:00
neha_gupta
6c2b78de07
Item size- sum of attachment and email body (#3291)
<!-- PR description-->

Size of emails will be - sum of - size of attachment and size of email body
In case of contacts and events, since mostly everything is data we will check the size as - total serialised bytes 

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix

#### Issue(s)
* #3152

#### Test Plan
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-10 02:34:12 +00:00
ashmrtn
f5365f19c5
Return map instead of using in-out param (#3308)
Return type should really be switched to []data.BackupCollection but that's a non-trivial change to make since tests rely on being able to lookup things by folder ID.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-05 21:00:16 +00:00
Abhishek Pandey
5b9cd69e29
Fetch calendar by name on 409 ErrorFolderExist (#3318)
Handle 409, `ErrorFolderExists` error in restore path while creating destination calendar. We need to fix this for all m365 services. This PR is focused on calendar only.

Context:
`CreateCalendar()` may fail with ErrorFolderExists under certain error conditions. For e.g. consider below scenario.

1. `CreateCalendar()` does a POST to graph to create restore destination calendar but this fails with 5xx.
2. It's possible that step 1 may have left some dirty state in graph. For e.g. it's possible that the destination folder in step 1 was actually created, but 5xx was returned due to other reasons.
3. So when we reattempt POST in such a scenario, we sometimes observe ErrorFolderExists error .
4. Corso should be resilient to such errors. To fix this, when we encounter such an error, we will do a GET to fetch the restore destination folder and add it to folder cache.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
Integration tests and mock tests will be added in a follow up PR. 
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-05 19:43:07 +00:00
Keepers
3a2d0876dd
consolidate aggregation of parent-item exclude map (#3258)
introduces a new type wrapping a nested map so that aggregation of globally excluded items in driveish services don't need to manage the map updates themselves.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2340

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-04 19:32:47 +00:00
ashmrtn
07b5acb92d
Temporary solution to duplicate Exchange folders (#3300)
For Exchange email or contacts backup, pick on the folder with
the highest ID in the hopes that it will be the most recently
used one

A better solution for this situation should be coming soon

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3197

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-04 01:51:53 +00:00
Keepers
c152ce8b25
move and rename common/time.go (#3215)
moves common/time.go to common/dttm/dttm.go, and renames many of the consts and funcs with names that better explain their design and intent of usage.

Only code movement, no logical changes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-02 19:39:29 +00:00
Keepers
e72fa49018
retry on gateway timeout (#3271)
We don't currently retry on graph API gateway timeout/bad gateway responses.  This change adds those statuses to our retryable codes.

Also adds some qol clues tracking and error formatting.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-02 17:47:45 +00:00
ashmrtn
dcb9d81f3f
Fix progress folder display for Exchange (#3219)
---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3013

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-29 00:19:54 +00:00
Keepers
edef23bfc4
errant cleanups before incremental support (#3201)
a handful of errant cleanups to help the PR for adding incremental support to sharepoint.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3136

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-25 21:09:41 +00:00
Keepers
0ae1a04097
use nanosecond folders for testing (#3205)
we've been failing on folder name collision, particularly in this package.  Probably because it's using second-granularity folder names.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-25 20:33:39 +00:00
Abhishek Pandey
c38a431079
Add --disable-concurrency-limiter cli flag (#3203)
<!-- PR description-->
The concurrency limiter middleware is enabled by default for exchange backups. Adding a hidden --disable-concurrency-limiter flag to disable this middleware.

Reasons for adding this flag:
We have done limited perf testing with the concurrency limiter. In addition, our understanding of exchange concurrency limits is also a bit fuzzy. This flag acts as a killswitch in case we start to see perf regressions in prod.

I see this as a temporary flag. Ideally we would never have to use it. Also, once we have a better way to configure concurrency limiter, we can eliminate this flag.


---

#### Does this PR need a docs update or release note?

- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature


#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->

- [ ]  Unit test
2023-04-25 01:22:12 +00:00
Abhishek Pandey
41f742eba2
Add concurrency limiter middleware package (#3182)
<!-- PR description-->


Graph service only allows [4 concurrent requests per exchange
mailbox.](https://learn.microsoft.com/en-us/graph/throttling-limits#outlook-service-limits)
We are currently not honoring this limit in corso. This causes 429
errors with “Application is over its MailboxConcurrency limit”. This PR
introduces a concurrency limiter middleware. This middleware is
selectively added for exchange backups only.


---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E

---------

Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
2023-04-24 11:16:20 -07:00
Keepers
62daf10213
migrate onedrive using prefix collection (#3122)
#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-24 17:36:10 +00:00
Keepers
6405c8246e
move parallelism logs/checks upstream (#3003)
The fetch paralellism checks and logs occur on
every item streamed from GC.  This is a bit chatty,
and has been moved upstream in the process for
a more centralized behavior.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-04-21 20:23:41 +00:00
Keepers
d5fac8a480
custom http wrapper for item downloads (#3132)
the http client customized by the graph client for use in downloading files doesn't include our middleware like expected. This introduces a new http client wrapper that populates the roundtripper with a middleware wrapper that can utilize our kiota middleware in the same way as the graph client does.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3129

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-20 23:20:23 +00:00
ashmrtn
f01e25ad83
Redo folder generation in backup details (#3140)
Augment the folder backup details entries with
LocationRef and some information about what
data type generated the entry. Also add top-level
container information like drive name/ID if
applicable

Refactor code to do folder generation to make it
more contained and require less parameters to add
entries to backup details

Important changes are in details.go. All other
changes are just to keep up with the slightly
modified function API or update tests

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3120
* closes #2138

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-19 20:27:55 +00:00
ashmrtn
ba724ac63d
Add immutable ID header to Get requests (#3157)
This allows using immutable IDs for Exchange objects. It does not add this header to put requests as we don't record the IDs of restored objects anywhere.

May slightly increase performance of events backups as
it now sets the delta query page size to 200 like with
other requests

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2861

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-19 16:16:49 +00:00
Keepers
86f9d60bf7
relocate connector mocks (#3048)
It makes more sense for mock data types to be
owned by the service package, not by the connector package as a whole.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-17 17:54:19 +00:00
Keepers
c90f587769
clean unused options, reduce folders call (#3127)
Cleans out some unused exchange options builders.
Also tops the mailFolders userinfo check to a single folder, since we only need one result to verify access.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-17 17:22:27 +00:00
Keepers
482b690e06
fix overlapping bases after fallback union (#3118)
In a case where the current manifests and the
fallback contain both distinct and overlapping
categories, the end result contains multiple
bases for the same category.  Verifydistinctresons
didn't catch this because it includes the resource owner.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
2023-04-14 16:29:29 +00:00
Abin Simon
6c64eb0c16
Fetch individual attachments in case of full download failure (#3086)
Failure can happen as the list of attachment and they data can get
pretty big.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-13 05:51:58 +00:00
Abhishek Pandey
ec064e539b
Add misc typo fixes (#3006)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?


- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->

- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
2023-04-12 23:30:38 +00:00
Keepers
da8ac5cdbc
moving code around, no logic changes (#3025)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
2023-04-12 17:16:05 +00:00
Keepers
30c86797d3
always include delta metadata, additional logging (#3092)
Ensure exchange always includes the delta metadata entry, even if it has zero deltaURLs to record.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* fixes #3019

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-12 00:43:55 +00:00
Abhishek Pandey
2300727b09
Sanitize recurrenceTimezone prior to restoring calendar event (#3064)
<!-- PR description-->
* Do not specify recurrenceTimeZone if it's not set in the item being
restored. "" is not a valid value
---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->

- [x] 🐛 Bugfix


#### Issue(s)
* COR-53

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💚 E2E
2023-04-10 13:17:53 -07:00
ashmrtn
59e17757bf
Switch LocationPath return type (#3038)
LocationRefs don't need prefix directories, so using a path.Path is
overkill. This will also fit better with changes to come where we need
to parse and update the LocationRef of merged backup details entries.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-05 21:10:42 +00:00
Keepers
fac5a0f3d2
No warn logs (#3024)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
2023-04-05 17:43:02 +00:00
Keepers
74e4a094e0
conceal url pii (#3015)
Adds a new package to common (pii) with a clues.
Concealer compliant struct that can log and report
url values with a mix of hidden pii and exposed
public values.  Exposed values are based on
well-known graph api path and query params.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
2023-04-04 21:01:37 +00:00
Keepers
fbb867a08d
add clues.Hide to various logs (#2994)
Begins hiding PII with clues.Hide injections in our structured logging aggregators.  This is a Non-
comprehensive change, and improvements here will be ongoing.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-04-04 16:59:31 +00:00
Abin Simon
4e1a44fbee
Remove unnecessary indirection for retries (#3021)
This was previously used to add retry logic which is no more the case as is a completely unnecessary indirection.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-04 16:05:25 +00:00
ashmrtn
16d1e68ebf
Populate location for all Exchange data types (#2829)
Always populate the location field of backup details
for Exchange data types

Fix bug where ParentPath for calendar items used
folder IDs instead of display names

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486
* closes #2827

merge after:
* #2826
* #2808
* #2828

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-03 20:22:31 +00:00
Keepers
f2bf0ee685
introduce fail-after-recovery option (#3005)
While fail-fast and best-effort make for good
categories at the extreme ends of error handling, 
we keep finding outselves wanting to operate
in a middle ground.  This change introduces a new
error handling category: FailAfterRecovery.  This
option tells corso to complete as much of its
process as it can, even if it recovers from errors.
But at the end of processing, if it recovered from
any errors, an error is returned for the operation.

This behavior is the new failure handling default,
instead of failFast.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-04-01 00:43:52 +00:00
Keepers
b81a527c10
add clues.Hide to replace observe.PII (#2993)
The observe PII handlers are getting replaced by
the more standardized clues secrets.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2024

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-03-31 22:14:09 +00:00
Keepers
25cffcdf8d
create a new http client on every delete (#2970)
in order to avoid errors cascading from graph API's
populated 204 responses containing `0/r/n/r/n`,
in particular in response to deletes, this change
hacks in the creation of a unique http client for
each delete call.

This is dangerous to do, since we already know
that generating clients per calls can leak
consumption of system resources such as sockets.
This change presumes that runtime deletes are
minimally, if ever, called during standard backup
and restore.  The permission deletion in onedrive
is the stand-out exception that might need some
additional investigation.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2707

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-30 15:56:53 +00:00
Keepers
e09c120778
add linter for using clues to create and wrap errs (#2959)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
2023-03-29 21:16:05 +00:00
ashmrtn
91e0a27831
Populate location in Exchange container resolver (#2828)
Always generate both a path of IDs and a path of
display names for folders in the container resolver.

Changes in internal/connector/exchange/service_functions.go
keep the behavior of the overall system from changing

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

merge after:
* #2808

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-29 01:19:59 +00:00
Keepers
940892fc07
correct input aliasing (#2960)
Updates and corrects input aliasing according to
the following rules (in priority order):
1. if the library name is usable, use it
2. if not, alias to the package name
3. if the package name is weird, alias sensibly
4. in case of collision, alias more distant imports
5. aliases should be consistent throughout

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
2023-03-28 20:48:22 +00:00
Keepers
9d73d0c8e2
replace errors.* with clues.* (#2924)
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.

Might follow up with a linter to enforce this change.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-28 04:01:54 +00:00
Keepers
328d75945b
PoC for centralized throttling control (#2887)
PoC for using a centralized middleware to control
1k-per-minute get requests to graph api.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💪 Manual
2023-03-24 00:40:08 +00:00
Keepers
b82c994b91
Always make empty collections for prefix directories (#2929)
Use empty collections to ensure the prefix directories are available in
kopia no matter what user data we find.

This helps ensure we have some set of things we can assume about
non-failed backups including:

    They have a valid snapshot ID for data
    They have a valid snapshot ID for backup details

Eventually, this will allow us to say that backups using a base that
doesn't have a prefix directory is invalid in some way

SharePoint tests are disabled because lists and pages ignore the none
target in selectors

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #2550

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-24 00:04:03 +00:00
ashmrtn
6f99bba624
Minor cleanup to mail resolver fetch (#2808)
Remove old code that explicitly fetched the Inbox. The issue in Graph SDK that was causing this is no longer present.

Retrieval of Inbox is tested by [test](41f0b86073/src/internal/connector/exchange/data_collections_test.go (L235)) for making collections

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-23 15:27:21 +00:00
Abin Simon
7a7933e54c
Assume that item has no attachments if there is no body (#2911)
Some response from the Graph API was returning empty body. This could be something left over from older version of the platform, but this causes issues in our codebase.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-22 22:53:45 +00:00
Keepers
e77a5e5a7e
additional logging when creating backup model (#2916)
#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2913
2023-03-22 20:00:33 +00:00
Keepers
532c5c017b
update backup cli output (#2903)
changes a couple things in the backup cli output:
* appends the backup id after completion
* prints an error if the backup fails
* replaces multierr with a clues error
* logs each backup failure individually

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1970

#### Test Plan

- [x] 💪 Manual
2023-03-22 16:01:02 +00:00
Abin Simon
ef5178668f
Log user name when enumerating drive items (#2872)
This mimics the behavior for exchange where we print out which user and what content is currently being enumerated. While this might not be useful when we are only backing up a single user, it can come in handy when we run backups for multiple users.

<!-- Insert PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2871

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-21 03:51:16 +00:00