717 Commits

Author SHA1 Message Date
ashmrtn
b6a7095c7f
Don't retry if an item was not found (#2233)
## Description

When fetching item data, don't backoff and retry if Graph reported the item was not found. In this case, we mark it as succeeded as we can never get the data anyway.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2217

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-23 18:35:16 +00:00
Keepers
b3d4b4687b
discovery api, filter guest and external users (#2188)
## Description

Adds the api client pkg pattern to the connector/
discovery package.  Most code changes are plain
lift-n-shift, with minor clean-ups along the way.

User retrieval is now filtered to only include
member and on-premise accounts.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #2094

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-01-20 20:09:59 +00:00
ashmrtn
bb7f54b049
Update OneDrive metadata with folder moves (#2193)
## Description

Track folder moves and update the persisted metadata accordingly. Moves need to take into account subtree changes because OneDrive will not notify us of subfolders moving if nothing else was updated on the subfolder.

Handle updates by making a copy of the folder map for ease of updating and later on ease of finding out which collection paths have changed

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2120

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-20 19:18:59 +00:00
ashmrtn
6651305d4f
Update OneDrive metadata with folder deletions (#2192)
## Description

Track folder and package deletions in OneDrive metadata for incremental backups. Add test for deletions

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2120

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-20 17:02:23 +00:00
ashmrtn
5d20e30062
Record graph error (#2195)
## Description

Record information about the error Graph returned if something went wrong when fetching an item.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2194

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-20 05:55:11 +00:00
ashmrtn
38b9fabb51
Check for and ignore item not found errors (#2202)
## Description

Item not found errors may appear because an item was moved or deleted between enumerating the items in a folder and fetching the data for the items in the folder. This keeps them from being reported as there is nothing we can do if the item data is no longer available.

This applies to all Exchange data categories

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2198

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-20 04:34:48 +00:00
ashmrtn
298ee35f05
Use mutex when appending to error (#2201)
## Description

Make error appending thread-safe since multiple goroutines may attempt to add an error at the same time.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2197

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-20 04:16:39 +00:00
Danny
4473215882
GC: Backup: [FIX]Embedded image requires download (#2189)
## Description
Images copy/pasted into a message were not being saved previously.  Verified with Corso binary
<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  Yes, in the ChangeLog.


## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2163<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x]  Unit test
2023-01-20 00:32:18 +00:00
Keepers
8df00bd386
standardizes the calendar resolver interface (#2162)
## Description

Adds a little extra process into the calendar
resolver so that it mimics the mail and contact
resolvers.  This will allow us to collapse the
three resolvers into a more common handler
or interface.

We can take this change or drop it.  I added
the code in exploration of the event test failures,
and figured I'd throw it out just in case.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2022

## Test Plan

- [x] 💚 E2E
2023-01-19 22:30:14 +00:00
Abin Simon
60a04c9ca4
Handle ResourceNotFound error when fetching drive (#2161)
## Description

Follow up for https://github.com/alcionai/corso/pull/2156. We might deice to change where we add the logic to handle this, as in if we should move it to discovery module.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2145
* https://github.com/alcionai/corso/pull/2156

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-19 07:49:07 +00:00
Danny
70ebaa6b0d
MetaData: Pages Backup Info Support (#2084)
## Description
SharePointInfo extended to include SitePageable objects. 
**NOTE:** Merge includes the beta graph package and will affect the compile time. 
<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes  #2069<issue>
* related #2071

## Test Plan

- [x]  Unit test
2023-01-19 04:06:30 +00:00
Danny
1db101ca17
GC: Restore: SharePoint add case for Page Category in Workflow. (#2114)
## Description
Fix to allow selectors to include `SharePoint.Pages` during backup without causing backup workflow to fail. 
<!-- Insert PR description-->

## Does this PR need a docs update or release note?
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #2113<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-01-19 03:42:31 +00:00
Danny
936789b6b4
GC: Backup: Sharepoint libraries to omit site drive. (#2098)
## Description
Branch treats Drive `Site Pages` as a restricted directory. All  `pages` within the directory, will be backed up via the Site Pages API.  This adds an additional call to per collection to obtain the parent `driveName`. 
<!-- Insert PR description-->
TODO: Create a Pipeline that distinguishes between SitePages, Libraries, and List for SharePoint
## Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR
The documentation should state the assumption is that only `.aspx` files are to be found in the s
## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2072<issue>

## Test Plan
- [x]  Unit test
2023-01-19 03:19:15 +00:00
Keepers
51e29f2975
use delta queries for calendar events (#2154)
## Description

Hacks the beta version call into the events api
when iterating through items, allowing us to run
delta-based queries for calendar events.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #2022

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-19 01:30:56 +00:00
Keepers
e3b6d035fb
extra loggng and error wraps (#2155)
## Description

Adds info logging on all throttling responses,
and some extra error handling in container
resolvers.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Test Plan

- [x] 💪 Manual
2023-01-18 16:43:19 +00:00
Danny
dc17c68074
Middleware: Context timeout expansion (#2048)
## Description
Adds additional definitions for context timeouts for middleware usage
<!-- Insert PR description-->

## Does this PR need a docs update or release note?


- [x]  No 

## Type of change

- [x] 🐛 Bugfix


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*related  #2047<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-01-18 14:08:00 +00:00
Vaibhav Kamra
3a37584938
Check whether the user has an exchange mailbox (#2156)
## Description

This commit adds logic in discovery and backup to check whether the specified user has
an exchange mailbox that is available/enabled.

If so - the backup is short-circuited to succeed but with "no data"

Going forward - we should be able to move the logic in the OneDrive connector that checks
for a valid drive and license in here.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2145 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-18 06:59:29 +00:00
ashmrtn
45874abf7e
Begin persisting OneDrive/SharePoint library metdata (#2144)
## Description

Start persisting the folder path maps and delta URLs for backed up OneDrive/SharePoint drives. Delta URLs are saved in a map[drive ID]deltaURL while folder IDs are in a map[driveID]map[folder ID]folder path

Needs another patch to properly save the path for folders that match the selector, currently the selector comparison is only on the parent of an item

Later PRs can get the new folder map by taking the map from the previous backup and making changes to it when folder deletions/moves are encountered

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2120 

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-17 20:50:24 +00:00
Keepers
1ef300d6c2
add item fetching and serialization to api (#2150)
## Description

Adds the per item collection streaming calls to
the api interface.  Primarily migrates a "getItem"
and a "serializeItem" acton into the api pkg, out
from exchange_data_collection.  Building an
ExchangeInfo is now also housed in api.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1996

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-17 20:27:37 +00:00
ashmrtn
8d798d6024
Switch to "delete wins" merge strategy for Exchange IDs (#2142)
## Description

#2116 implemented a "last change wins" strategy for deduping IDs. However, it appears Graph API doesn't really enforce an ordering for returned items so that strategy may not work. Instead, implement a "delete wins" strategy.

Items that are added, deleted, and then restored in M365 will be restored with a different ID

This PR does the following:
* reverts changes to return a unified list of IDs and added/removed status
* switches to a "delete wins" merge strategy for duplicate IDs
* updates tests

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #1954 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-13 20:11:25 +00:00
Keepers
5980c4dca0
log observed messages (#2073)
## Description

add logging to the observe package, assume that
every instance where a message is observed, it
also gets logged.

Merger may want to wait until logging to a file is the standard behavior, else the terminal might get messy/buggy.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2061

## Test Plan

- [x] 💪 Manual
2023-01-13 19:34:01 +00:00
ashmrtn
f43c458844
Dedupe collection item IDs prior to fetching item data (#2116)
## Description

Under some conditions, items can be returned multiple times when enumerating the items in a folder in Exchange. Begin deduping those items by doing the following:
* use a map in ExchangeDataCollection to dedupe on item IDs
* have enumeration function return a single slice of items containing both additions and deletion so a "last-change-wins" policy can be implemented

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1954 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-13 02:19:21 +00:00
ashmrtn
edd08269f6
Reenable OneDrive kopia-assisted incrementals (#2137)
## Description

Reenable kopia-assisted incrementals now that #2136 contains the bugfix patch

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2133

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-13 00:35:45 +00:00
Vaibhav Kamra
5d70b4b3a6
Disable kopia incrementals for OneDrive (#2132)
## Description

Kopia-assisted incremental backups do not appear to have the correct file size set on restore

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-12 07:40:49 +00:00
Vaibhav Kamra
cfe6549987
Re-Enable Kopia-assisted incrementals for OneDrive (#2126)
## Description

This addresses the deadlock in the item progress reader by deferring the reader creation to
when the first read is issued for the item

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1702 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-12 03:50:26 +00:00
ashmrtn
32b11cdca8
Dedupe collection items for OneDrive (#2118)
## Description

Under some circumstances items can be returned multiple times when iterating through the endpoint. This dedupes items within a single collection.

It does not address the following:
* item being removed from the collection during iteration
* item appearing multiple times in different collections

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1954 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-12 00:00:41 +00:00
Danny
9f05e83d43
CLI: Adds to the displayable headers for OneDrive and SharePoint. (#2090)
## Description
The update adds `DriveName` to SharePoint details. Values are currently the M365ID

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #1938<issue>
* related to #2064 


## Test Plan
- [x]  Unit test
2023-01-11 12:49:20 +00:00
Vaibhav Kamra
fc17c387f1
Update list of skus that license onedrive (#2105)
## Description

Adds a few missing SKUs that license OneDrive. This is not the complete list.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2104 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-11 07:09:14 +00:00
Keepers
580934b069
remove DiscreteScopes() from selectors (#2036)
## Description

DiscreteScopes is a vestigial func from when scopes contained the list of resource owners to track.  That behavior is no longer in use.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
2023-01-10 23:24:01 +00:00
Keepers
1174a99e84
refactors the exchange fetchIDs iters (#1906)
## Description
    
Transitions the fetchIDs service iterators to a
set of interfaces to consolidate code across
multiple nearly identical variations of "fetch id
for directory".

This was originally written in another PR (1780), then
separated out to isolate changes.

## Type of change

- [x] 🐹 Trivial/Minor

## Test Plan

- [x] 💚 E2E
2023-01-10 21:31:20 +00:00
ashmrtn
778a60774a
Fix unit tests that fail when not running with env vars (#2089)
## Description

Refactoring and new code additions have led to some unit tests that fail unless testing is run with the proper env vars. Fixup code to either skip those tests or provide them with the proper information so they pass

* some tests moved from unit tests to integration tests
* function to get credentials with bogus info for unit tests
* skip some tests if env vars not passed

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2052 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-10 19:45:02 +00:00
Danny
85b5877987
Backup Details: Onedrive details expanded (#2085)
## Description
Updates the amount of metadata available for OneDrive backups.  This does not affect the way `OneDrive` information is displayed to the user. 
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #2074<issue>

## Test Plan

- [x]  Unit test
2023-01-10 18:56:38 +00:00
Keepers
bf0a01708a
remove selector printable, list discreteOwner (#2034)
## Description

The selectors printable code wasn't being used in any valid way, so that's out.  Backup List printing now refers to the DiscreteOwner in the embedded selector as the ResourceOwner reference.  Renamed the "Selectors"
column in the List print table to "Resource Owner" to better match what should be contained in that field.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🐛 Bugfix
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1617

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-01-09 22:42:17 +00:00
Keepers
059b860fde
Invert incremental flags, default to incremental (#2051)
## Description

Sets the default run behavior for exchange to use
incremental backups.  The cli feature flag for enabling exchange incrementals has been swapped for a toggle that disables incrementals, forcing a full backup

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1901

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-08 20:22:56 +00:00
Keepers
d8763298ce
test service_iterators with mocks (#2020)
## Description

Now that we have a mocked api layer in place, we can test filterContainersAndFillCollections without
integration dependencies.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
2023-01-06 22:19:45 +00:00
Danny
b769c288b7
[Bug Fix] GC: Backup: Onedrive: Context timeout during drive retrieval fix (#2046)
## Description
Adds manual retry for drive retrieval <!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix


## Issue(s)

* closes Issue  #2044<issue>

## Test Plan

- [x]  Unit test
2023-01-06 16:17:21 +00:00
Keepers
2b45cfa617
remove resource owners from scopes (#1895)
## Description
    
Now that resource owners are identified via
the selector itself, rather than each scope, we
can remove the resource owner data from
scope production and data.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-05 21:29:48 +00:00
Keepers
70d5a5ab56
transition api methods to interfaces (#2010)
## Description

replaces the new api client methods with interfaces, to prepare for testing funcions with mocks instead of integration.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-05 21:03:35 +00:00
Danny
4bcfc9dadf
GC: Restore: SharePoint List Workflow (#1964)
## Description
Connects restore pipeline for List Category. CLI commands are not included, Documentation for List restore needs to be placed within the CLI PR
<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

## Issue(s)

* closes #1962<issue>
* closes #1978
* closes #1935

## Test Plan

- [x]  Unit test
2023-01-05 14:59:48 +00:00
Keepers
1f26339813
Match resource owners at top of reduce (#1891)
## Description

Checks for resource owner matches in the top
of the reduce func using the selector owners,
instead of waiting until the path match check.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-05 00:59:16 +00:00
Keepers
edc4426b9c
Fix --users "*' not finding user in tenant (#2033)
## Description

DataCollections validation step was still using the full resourceOwner list in the selector to validate every backup, rather than checking only the DiscreteOwner.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
2023-01-05 00:29:21 +00:00
Danny
bb5b2f23e9
GC: Framework to use adapter's serialization writer. (#2032)
## Description
The MS Graph's adapter contains a serialization writer within it. Create a framework to use this writer rather than creating the object within serialization workflows. The writer library was used separately when the package still had many bugs and the most current version was required. This does not change any of the writer instances within the package. Will change out the writer instances in follow-up PRs.

<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*related to  #2031 <issue>

## Test Plan
- [x]  Unit test
2023-01-04 20:59:13 +00:00
Keepers
672fe54c41
migrate resolver iterators (#2009)
## Description

Previous changes held out on migrating the
resolver iterators into the api package because
they embedded function calls from the resolvers
themselves.  This change migrates that code
into the api package, and accepts a callback
function to hook in the resolver updates.  This
change sets up resolvers to better utilize an
interface in the next PR.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 20:02:47 +00:00
Keepers
0d0a7516f0
use selector owners, not scope owners (#1890)
## Description

Migrates code away from pulling the resource
owner from each scope, and instead usees the
selector as the canon identifier of the resource
owner.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 19:39:25 +00:00
Danny
76984c23c7
Mock: SharePoint: List objects expansion (#2019)
## Description
Adds the ability to create non-empty lists within mocks
<!-- Insert PR description-->

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change


- [x] 🤖 Test
- [x] 💻 CI/Deployment

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes to  #2018<issue>

## Test Plan


- [x]  Unit test
2023-01-04 18:42:55 +00:00
Keepers
978f9304bf
methodize all api funcs to the Client (#2008)
## Description

In order to use the api layer as an interface, we
need the functions therein to be methods, so
that callers can leverage local interfaces.  This
change introduces the api.Client, and begins
to spread it throughout the exchange package,
largely in place of graph servicers.

No logic changes have occurred here.  The only
modifications are what is required to utilize the
api client.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 17:19:44 +00:00
Keepers
e5edbfd77c
first pass migrate graph api to subfolder (#2003)
## Description

In order to establish a standard api around our
graph client usage, and thus be able to mock
for testing, we need to migrate graph client
usage into another pacakge.  This is the first
step in that process of refactoring.

All changes are code relocation importing and
exporting may change as needed.  No logic
was altered.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 01:35:32 +00:00
Keepers
1586ed927b
remove 0-item progress bars (#1986)
## Description

Incremental backups can display progress bars
with 0 items to add or remove.  This can cause
end-user confusion when hte bar completes
with 0 items involved.  We should instead keep
those bars hidden.

Also removes the "heuristics" progress step.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1984

## Test Plan

- [x] 💪 Manual
2023-01-03 23:22:20 +00:00
Keepers
ce1f4406f1
No tester errors 2 (#2005)
## Type of change

- [x] 🐹 Trivial/Minor
2023-01-03 21:35:56 +00:00
Keepers
07faa7bffb
split selectors on discrete resource owners (#1889)
## Description

Switches the CLI from calling `DiscreteScopes` to `SplitByResourceOwner`
on the selector itself.  This func will take the original selector and produce
a slice of selectors, each one with a DiscreteOwner (the single user involved
in usage of that selector) and all include/filter scopes in that selector re-rooted
to that discrete owner.

Does not yet solve the per-category tuple, since we are still pivoting on the
scopes inside the selector.  That comes as a later change.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-03 20:55:39 +00:00