1098 Commits

Author SHA1 Message Date
ashmrtn
ba724ac63d
Add immutable ID header to Get requests (#3157)
This allows using immutable IDs for Exchange objects. It does not add this header to put requests as we don't record the IDs of restored objects anywhere.

May slightly increase performance of events backups as
it now sets the delta query page size to 200 like with
other requests

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2861

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-19 16:16:49 +00:00
ashmrtn
9664afaa12
Restore with OneDrive/SharePoint ID (#3151)
Allow restoring OneDrive and SharePoint items by
M365 ID as well as file name and ShortRef

Does create an edgecase where attempting to
restore an item named the same as another item's
M365 ID causes both items to be restored

Moves the consts for OneDrive file suffixes to a
different package so that the selectors package
can also import them without a cycle

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3142

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-19 15:30:03 +00:00
Keepers
d5eee4479d
move largeItem client constructor into new GC (#3130)
It's an unnecessary paramter.  Plus, this sets up the next pr to have a smaller footprint.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3129

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-18 18:52:22 +00:00
Keepers
0458d04b9e
use slices for all filter targets (#3028)
Simple concept for a large footprint of change:
filters should always accept a slice of targets
to compare against, instead of a string.  Currently,
we're faking a slice by way of using a const
separator anyway, so we might as well support
slices outright.

Lots of changes (simplification, mostly) in
selectors cascade from this shift.  Any updates
you see here are purely to comply with the updated
filters interface.

This step will make concealing PII in selector and filter
logging significantly easier than before.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
2023-04-17 22:07:17 +00:00
Keepers
86f9d60bf7
relocate connector mocks (#3048)
It makes more sense for mock data types to be
owned by the service package, not by the connector package as a whole.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-17 17:54:19 +00:00
Keepers
c90f587769
clean unused options, reduce folders call (#3127)
Cleans out some unused exchange options builders.
Also tops the mailFolders userinfo check to a single folder, since we only need one result to verify access.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-17 17:22:27 +00:00
ashmrtn
1826b66e8d
Populate and persist OneDrive LocationRef (#3111)
Leverage structs from previous PRs to
populate and persist the LocationRef
for OneDrive

Right now the source of the
LocationRef is still based off the
RepoRef path, but this is in the
OneDrive code only

Manually tested
* updating a subfolder path when parent folder was moved
* populating LocationRef when the base backup didn't have LocationRefs when nothing was changed in OneDrive
* populating LocationRef when the base backup didn't have LocationRefs when something was changed in OneDrive

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2486

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-14 21:31:16 +00:00
Keepers
b0869fad4d
add sensitive info flag to cli (#3000)
The sensitive-info flag allows CLI users to specify the Concealer algorithm used to handle PII and
other sensitive info in logging.  Default alg is 'hash', which produces a truncated hmacsha256 string.
'mask' for flat '***' replacement and 'plaintext'
are also available.

SDK consumer are expected to configure clues
themselves, rather than specifying a config option.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2024

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-04-14 17:07:14 +00:00
Keepers
482b690e06
fix overlapping bases after fallback union (#3118)
In a case where the current manifests and the
fallback contain both distinct and overlapping
categories, the end result contains multiple
bases for the same category.  Verifydistinctresons
didn't catch this because it includes the resource owner.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
2023-04-14 16:29:29 +00:00
ashmrtn
a25948042a
Wire up LocationIRef stuff to details merge (#3110)
This still doesn't populate LocationRef for
OneDrive, but gets a lot of the other logic
that will be required for that in place.

Mostly it focuses on starting to use
LocationIDer for lookups when merging
details

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-14 00:48:12 +00:00
Keepers
43c9ad4a01
clean up cli e2e tests (#3068)
deduplicate some e2e boilerplate setup.
Also, transitions exchange e2e tests from an in-
test loop over categories, to a well-defined test
func per category.  The former design doesn't
play well with go test --run ./testName, and
causes all tests to run even if they don't match
the string.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💚 E2E
2023-04-13 07:15:30 +00:00
Abin Simon
6c64eb0c16
Fetch individual attachments in case of full download failure (#3086)
Failure can happen as the list of attachment and they data can get
pretty big.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-13 05:51:58 +00:00
Abhishek Pandey
ec064e539b
Add misc typo fixes (#3006)
<!-- PR description-->

---

#### Does this PR need a docs update or release note?


- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->

- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
2023-04-12 23:30:38 +00:00
ashmrtn
40b605712d
Fix NPE panic (#3109)
Failing [test](https://github.com/alcionai/corso/actions/runs/4681548110/jobs/8294288107?pr=3107#step:7:879)

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3108

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-12 19:41:47 +00:00
ashmrtn
9e692c7e2e
Create abstraction to represent a unique location (#3100)
UniqueLocation allows representing a folder path that has
both a location that will be stored in backup details and
a location that can be used as a key in maps. The key for
maps is guaranteed to be unique for all data types within
a service

Add a function to extract a unique location from a backup
details entry and tests for that

UniqueLocation will eventually be used in GraphConnector
and KopiaWrapper so it needs to be in a package that both
of them can import

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-12 18:08:46 +00:00
Keepers
da8ac5cdbc
moving code around, no logic changes (#3025)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
2023-04-12 17:16:05 +00:00
Abin Simon
b1606466c2
Compute parent permissions from available metadata (#3022)
Previously we were querying Graph to get the current permissions and then using that to compute the permission diff. This changes that to use the metadata that we have available locally to do the computation avoiding a graph call to get permissions.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2976

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-12 12:22:35 +00:00
ashmrtn
52e627189a
Combine backup details merging structs (#3056)
Combine the sets of information used for merging
backup details so there's fewer things to pass
around and we can hide the function used to generate
lookup keys

This changes the prefix matcher to act like a regular
map for the moment (exact match) though OneDrive
LocationRef PRs will update that

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-12 03:08:40 +00:00
ashmrtn
8867b63ccb
Fixup manifest selection with fallback logic (#3097)
Fixup errors in manifest search logic that would cause
Corso to fallback to a full backup

* don't request fallback manifests if the user display name is the same as the user ID while Corso transitions to using user IDs
* dedupe manifests that are selected for multiple reasons

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3089

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-12 01:44:53 +00:00
Keepers
1f8c7598b4
separate fault bus in fetch manifests (#3090)
The global aggregation of recoverable errors
causes us to mark operations as failed even in
conditions where they should have succeeded,
because higher level callers don't have sufficient
control over ignorable error conditions.  This
is a short-term fix that allows us to log and dump
fault error accumulation in a specific, scoped instance.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-12 01:16:40 +00:00
Keepers
30c86797d3
always include delta metadata, additional logging (#3092)
Ensure exchange always includes the delta metadata entry, even if it has zero deltaURLs to record.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* fixes #3019

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-12 00:43:55 +00:00
ashmrtn
ab3011cab4
Log resource owner name, not full selector during backup (#3098)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3096

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-12 00:05:39 +00:00
Keepers
50e92b65c6
add Concealer compliance to paths (#3017)
Adds compliance with clues.Concealer to the paths
package.  Also introduces a new struct in the same
space: Elements, which is a thin wrapper around
a slice of strings so that subsections of a path
or builder can carry the same pii behavior
without additional work on the consumer's end.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🌻 Feature
- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-11 22:04:34 +00:00
Keepers
10a5d21cc9
remove "errors" from log message (#3087)
#### Type of change

- [x] 🤖 Supportability/Tests
2023-04-11 19:26:03 +00:00
ashmrtn
cae3debd28
Lint for using suite.T().Run instead of testify's suite.Run (#2664)
Includes fixups for calls that need updated

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [x] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2373

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-11 16:39:39 +00:00
Keepers
21cd7210cb
remove weburl suffix support from sharepoint (#3082)
We're dropping support for selection-by-url-suffix in sharepoint.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-11 16:09:58 +00:00
ashmrtn
2ebab1a78b
Use prefix matcher when merging backup details (#3055)
Store all locations in the prefix matcher
and then look them up when merging details

Intermediate step to get things setup for
having OneDrive locations during merging

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-11 03:23:13 +00:00
Keepers
7c9eada5a9
optionally look up and union fallback manifests (#3057)
Fallback manifests allow us to find manifests and
metadata from snapshots that utilized earlier
versions of kopia Reasons, primarily across
resource owner deltas.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-04-11 02:47:11 +00:00
Abhishek Pandey
2300727b09
Sanitize recurrenceTimezone prior to restoring calendar event (#3064)
<!-- PR description-->
* Do not specify recurrenceTimeZone if it's not set in the item being
restored. "" is not a valid value
---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->

- [x] 🐛 Bugfix


#### Issue(s)
* COR-53

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💚 E2E
2023-04-10 13:17:53 -07:00
ashmrtn
89ac00e64e
Add basic prefix matcher type (#3054)
Adds a basic generic prefix matcher type. The
first implementation just uses a map and iterates
through things

This is expected to be used for at least the
exclude list for backups (inner type of
`map[string]struct{}`) and logic for merging
backup details (inner type still TBD a bit,
but either `*path.Builder` or a custom struct)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-10 19:11:53 +00:00
ashmrtn
27909592b8
Exit early if context is marked as done (#2883)
Keeps the code from blocking and then failing immediately without sending a retry.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2882

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-10 18:38:14 +00:00
Georgi Matev
b6ae9c6d07
Switching user identification and hashing (#3071)
Encompasses the following changes:
* distinct ID for users and events set to m365 tenant ID hash
* still record local repoId
* switches the hashing method for the m365 tenant ID to tuncated sha256
* continue logging deprecated md5 hash for a few releases to facilitate event merge

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-10 16:44:23 +00:00
Keepers
896c05f623
look up users and sites by id or name (#2973)
Adds a lookup step to graph connector to find
an owner's id and name given some identifier.
The identifier, for either sites or users, can be a
well formed id or name.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-07 22:29:09 +00:00
Keepers
3cc29649ed
Add GetUserInfo to services/m365 (#2991)
Adds an interface for GetUserInfo, which gets
corso-specific metadata for a single user.  Also
does some refactoring around discovery for
better interface consistency, both as variables
and as access layers.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-07 00:36:50 +00:00
Keepers
0fd1314479
use clues with plaintext as default (#3058)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-06 23:15:20 +00:00
Abin Simon
baddd9fc83
Get OneDrive item with getM365 (#2791)
Add support for onedrive in getM365

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-06 05:32:03 +00:00
ashmrtn
59e17757bf
Switch LocationPath return type (#3038)
LocationRefs don't need prefix directories, so using a path.Path is
overkill. This will also fit better with changes to come where we need
to parse and update the LocationRef of merged backup details entries.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-05 21:10:42 +00:00
Keepers
0d521bb5a7
move sharepoint onto map-based lookup (#2958)
Refactors the sharepoint site lookup to use the
id-to-name maps.  This has a momentary regression
that will get solved in the next PR: we no longer
match on weburl suffixes, and instead require
a complete match.

Also, migrates the sharepoint lookup code out
of GC and into Discovery.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825
* #1995
* #1547

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-05 19:38:22 +00:00
Keepers
fac5a0f3d2
No warn logs (#3024)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
2023-04-05 17:43:02 +00:00
Keepers
74e4a094e0
conceal url pii (#3015)
Adds a new package to common (pii) with a clues.
Concealer compliant struct that can log and report
url values with a mix of hidden pii and exposed
public values.  Exposed values are based on
well-known graph api path and query params.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x]  Unit test
2023-04-04 21:01:37 +00:00
Keepers
2341d61842
introduce id-name lookup maps (#2955)
Adds two maps to resource-owner handling:
id-to-name and name-to-id.  Expectation is that
these maps will either get populated by a caller
as a pre-process before initializing the gc client, or
gc will (later pr) be able to look up the owner and
populate those maps itself.  The maps are
used to set the selector id and name for iface
compliance.  Only supported by exchange in this PR.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-04 20:21:58 +00:00
Keepers
fbb867a08d
add clues.Hide to various logs (#2994)
Begins hiding PII with clues.Hide injections in our structured logging aggregators.  This is a Non-
comprehensive change, and improvements here will be ongoing.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #2024

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-04-04 16:59:31 +00:00
Abin Simon
4e1a44fbee
Remove unnecessary indirection for retries (#3021)
This was previously used to add retry logic which is no more the case as is a completely unnecessary indirection.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-04 16:05:25 +00:00
Abin Simon
c696e7b7ff
Enable permissions backup by default (#2981)
Previously we had a flag that needed to be flipped to backup permissions. Now we always backup permissions.

- Should we wait till https://github.com/alcionai/corso/issues/2976 is done to do this
- Users will have to do a non incremental backup first as of now to get permissions. We should probably have some way to force this.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/2409

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-04 03:55:44 +00:00
Keepers
99774e754b
introduce idnamer (#2934)
Adds a common interface: idNamer, which is used
to pass around tuples of an id and a name for some
resource.  Also adds compliance to this iface in
selectors, where a selector's ID and Name are the
DiscreteOwner values.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-04-03 22:42:48 +00:00
ashmrtn
16d1e68ebf
Populate location for all Exchange data types (#2829)
Always populate the location field of backup details
for Exchange data types

Fix bug where ParentPath for calendar items used
folder IDs instead of display names

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486
* closes #2827

merge after:
* #2826
* #2808
* #2828

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-03 20:22:31 +00:00
Abin Simon
4cc0397075
Log responses for 503 (#3012)
Currently we were logging rate limiting values for 503 which is not available and end up with empty values. Switching it to logging the response, similar to 404 as that can be more useful.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-03 17:32:44 +00:00
Keepers
dcf5ea2415
update failure logs (#2998)
Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
2023-04-03 14:40:01 +00:00
neha_gupta
4087b69a39
Enhancement Sanity Test: verify permissions and defined data (#2938)
<!-- PR description-->
onedrive -  
- verify permissions - grantedToV2 and roles 
- recursively check subfolder's data and permission

Exchange -
- since the data in email is huge, create custom data and check if that data is getting restored properly
 
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-03 10:43:48 +00:00
Keepers
f2bf0ee685
introduce fail-after-recovery option (#3005)
While fail-fast and best-effort make for good
categories at the extreme ends of error handling, 
we keep finding outselves wanting to operate
in a middle ground.  This change introduces a new
error handling category: FailAfterRecovery.  This
option tells corso to complete as much of its
process as it can, even if it recovers from errors.
But at the end of processing, if it recovered from
any errors, an error is returned for the operation.

This behavior is the new failure handling default,
instead of failFast.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-04-01 00:43:52 +00:00