323 Commits

Author SHA1 Message Date
Abin Simon
9da0a7878b
Backup versioning (#2324)
## Description

Add backup format version information to the backups so that we can distinguish between backups which use a single file vs the ones that use both .data and .meta files.

Overrides https://github.com/alcionai/corso/pull/2297. I've also set it against `main` so that the diff shows up properly.
Ref: https://github.com/alcionai/corso/pull/2324#issuecomment-1409709118

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2230
* https://github.com/alcionai/corso/issues/2253

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual (Tested manually, will add an e2e test in a followup)
- [ ]  Unit test
- [ ] 💚 E2E
2023-02-03 04:03:14 +00:00
Abin Simon
b8bc85deba
Metadata backup for OneDrive (#2148)
## Description

This PR adds option to backup and restore additional metadata(currently permissions) for OneDrive.

**This PR also adds the ability to restore empty folders for OneDrive along with their permissions.**

Breaking change: Any old backups will not work as we expect both `.data` and `.meta`/`.dirmeta` files to be available for OneDrive backups.

## Does this PR need a docs update or release note?

*Added changelog, docs pending.*

- [x]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1774

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-02-03 03:25:51 +00:00
Keepers
0436e0d128
extra panic protection in operations (#2383)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Test Plan

- [x] 💚 E2E
2023-02-03 02:01:42 +00:00
Keepers
ec7d3c6fc5
add logging around operation completion (#2337)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2329

## Test Plan

- [x] 💪 Manual
2023-02-01 02:19:44 +00:00
Keepers
070b8fddee
update operation/backup errs (#2239)
## Description

Begins updating operations/backup with the new
error handling procedures.  For backwards
compatibility, errors are currently duplicated in
the old stats.Errs and the new Errors struct.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-31 22:25:02 +00:00
Keepers
498fc325a9
store errors in operation (#2237)
## Description

Adds the fault.Errors struct (now exported) to the
operations base.  stats.Errs is retained in the
backup and restore wrappers to avoid breaking
changes and allow for deserialization.  We will
continue to use the current error return until
dependencies are fully updated to use Errors.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-31 21:22:53 +00:00
ashmrtn
2458508969
Wire up GraphConnector <-> BackupOp item exclude list (#2245)
## Description

Return an item exclude list from GraphConnector to BackupOp. BackupOp does not yet pass this to kopia wrapper.

Returned list is set to nil (eventually) by all components so even if this were wired to kopia wrapper it wouldn't change the current behavior of the system

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2243

merge after:
* #2143 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-30 19:44:06 +00:00
Keepers
03642c517f
standardize deleteContainer func in exch api (#2246)
## Description

Minor refactor that came up while hunting bugs.
For the life of me, I cannot find the reason for
read/write counts being off in event incrementals.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2022

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-01-30 19:24:44 +00:00
Keepers
637b1904aa
remove operation "started" (#2247)
## Description

This flag is confusing, and is better represented by
tracking errors.

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Test Plan

- [x]  Unit test
2023-01-30 18:57:45 +00:00
Danny
d763d95976
BUG Fix: Restore: Operations (#2316)
BUG FIX: Adds clause to return an error if experienced during path building of a collection. 

## Description

<!-- Insert PR description-->

## Does this PR need a docs update or release note?
- [x]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
- related to #2257
## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-01-30 16:51:32 +00:00
Keepers
24911a358b
quick graph client refactor (#2301)
## Description

A couple minor refactors.  First, combines the
graph/service_helper.go and graph/service.go
files into a single file (just lift'n'shift, no logic
changes).  Second, renames CreateHTTPClient
to just HTTPClient.  Third, replaces the new
LargeItemClient constructor with extensible
options for the http client, starting with a
NoTimeout option.  This will provide longer
term maintenance over multiple constructors.

This change sets up QoL for building multiple services
and clients within api layer handlers, which will appear
in an immediately following PR.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2299

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-27 19:34:00 +00:00
ashmrtn
f69dfc14e2
Wire kopia exclude list up to BackupOp (#2244)
## Description

Expose the global exclude list mechanism in kopia BackupCollections to other components. Add tests for BackupCollections testing new exclude list functionality

Currently wired to always be passed nil

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2243

merge after:
* #2143 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-27 17:09:40 +00:00
ashmrtn
06efad7028
Bulk up kopia wrapper error checking in BackupOp (#2281)
## Description

Expand error checking for kopia wrapper backup. Kopia may report a number of errors for a snapshot operation but may not give error structs for them. Handle this by checking if there were > 0 errors reported and making our own error struct for this if there are.

This will keep the system from reporting a successful backup when kopia had problems reading or uploading data. Errored files in kopia do not become part of the snapshot

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2280 

## Test Plan

- [ ] 💪 Manual
- [ x  Unit test
- [x] 💚 E2E
2023-01-27 01:38:58 +00:00
Keepers
d529d145cb
scrub pii from observe logs (#2285)
## Description

This is a quick hack to satisfy a primary case of PII scrubbing.  We expect to revisit it in the future.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #2284

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-01-27 00:31:57 +00:00
Vaibhav Kamra
bdb7f2b109
Re-use client for OneDrive item downloads (#2276)
## Description

This PR reintroduces the changes from #2266 with a change to *not* reset the transport
when initializing the shared client. 

Doing so was removing the retry and other middlewares
and also resulting in throttled requests being masked as success

Also - we now decorate our download traffic with an ISV tag as recommended [here](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic)

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2266 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-01-26 15:05:12 +00:00
Vaibhav Kamra
e9354a8429
Revert "add itemClient for re-usable od item downloads" (#2273)
Reverts alcionai/corso#2266. Am seeing issues on data restore that need to be debugged.
2023-01-26 07:52:56 +00:00
Keepers
bda8a5c60c
add itemClient for re-usable od item downloads (#2266)
## Description

onedrive currently constructs a new http client
for every file it downloads.  This causes the OS
to generate extra sockets, and hang onto them
after the download is complete.  Replacing these
one-off clients with a singular, re-used client-
which is the behavior and standard suggested
for golang http clients- minimizes system
resource consumption.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* closes #2262

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-01-26 04:27:52 +00:00
Keepers
78b9f2752e
adds clues package to begin structured logging (#2214)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1969

## Test Plan

- [x] 💪 Manual
2023-01-25 21:39:38 +00:00
Keepers
e828209a30
return gc stats errors in backup (#2248)
## Description

If gc.stats reports a non-zero error count at the
end of a backup, retrieve the error from the
status and return it as the backup operation err.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-25 10:44:00 +00:00
Vaibhav Kamra
4a6959f60f
Revert "Issue 2022 eventincr" (#2256)
Reverts alcionai/corso#2167. This is causing the integration test to fail consistently (in CI and local runs)
2023-01-25 10:21:00 +00:00
Keepers
39c3c8a895
Issue 2022 eventincr (#2167)
## Description

Fixes and unskips the tests incremental integration tests
for exchange calendar events.

Currently blocked on LynneR displaying the calendar
fetch issues where it's only possible to retrieve her default
Calendar, Birthdays, and US Holidays calendars.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #2022

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-01-23 21:38:09 +00:00
ashmrtn
7f44db6c7c
Fail operation if we failed to fetch item data (#2199)
## Description

If GraphConnector reported errors while trying to fetch item data, fail the entire operation. This stops silent failures, but is a big hammer at the moment because there's no checks for items no longer being available.

Error reporting is minimal as well, but some info should be in the log.

**This effectively disables best-effort as any failure will result in the operation being reported as failed**

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2196

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-20 18:59:15 +00:00
Keepers
51e29f2975
use delta queries for calendar events (#2154)
## Description

Hacks the beta version call into the events api
when iterating through items, allowing us to run
delta-based queries for calendar events.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #2022

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-19 01:30:56 +00:00
ashmrtn
f01c8ad843
Add logging for bases in incremental backups (#2151)
## Description

Add log statements noting which bases were used for kopia assisted incrementals and which bases were merged into the hierarchy. Also record the reasons a base was chosen.

Log statements when searching for previous snapshots will be added when that code is refactored

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2149 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-17 21:51:57 +00:00
Keepers
48e4b65165
migrate, test manifest+meta producer (#2091)
## Description

Adds mocked unit tests for produceManifestsAnd-
Metadata.  For cleanliness, moves that func, and
any funcs called within it, to their own file within operations

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #2062

## Test Plan

- [x]  Unit test
2023-01-17 21:22:30 +00:00
ashmrtn
8d798d6024
Switch to "delete wins" merge strategy for Exchange IDs (#2142)
## Description

#2116 implemented a "last change wins" strategy for deduping IDs. However, it appears Graph API doesn't really enforce an ordering for returned items so that strategy may not work. Instead, implement a "delete wins" strategy.

Items that are added, deleted, and then restored in M365 will be restored with a different ID

This PR does the following:
* reverts changes to return a unified list of IDs and added/removed status
* switches to a "delete wins" merge strategy for duplicate IDs
* updates tests

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #1954 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-13 20:11:25 +00:00
Keepers
5980c4dca0
log observed messages (#2073)
## Description

add logging to the observe package, assume that
every instance where a message is observed, it
also gets logged.

Merger may want to wait until logging to a file is the standard behavior, else the terminal might get messy/buggy.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2061

## Test Plan

- [x] 💪 Manual
2023-01-13 19:34:01 +00:00
ashmrtn
f43c458844
Dedupe collection item IDs prior to fetching item data (#2116)
## Description

Under some conditions, items can be returned multiple times when enumerating the items in a folder in Exchange. Begin deduping those items by doing the following:
* use a map in ExchangeDataCollection to dedupe on item IDs
* have enumeration function return a single slice of items containing both additions and deletion so a "last-change-wins" policy can be implemented

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1954 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-13 02:19:21 +00:00
Vaibhav Kamra
cfe6549987
Re-Enable Kopia-assisted incrementals for OneDrive (#2126)
## Description

This addresses the deadlock in the item progress reader by deferring the reader creation to
when the first read is issued for the item

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1702 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-12 03:50:26 +00:00
Keepers
580934b069
remove DiscreteScopes() from selectors (#2036)
## Description

DiscreteScopes is a vestigial func from when scopes contained the list of resource owners to track.  That behavior is no longer in use.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
2023-01-10 23:24:01 +00:00
ashmrtn
e88553d073
Refactor the Reason struct and how tags for lookups are generated (#2087)
## Description

Solidify the interface between BackupOp and KopiaWrapper by making Reason the de facto way to pass/generate tags for things. The Reason struct now includes a function to generate tags for that instance. KopiaWrapper also now hides the fact that it prefixes tags from other components

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1916 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-01-10 20:32:17 +00:00
ashmrtn
43c2017492
Don't use OwnersCats struct (#2055)
## Description

Remove OwnersCats so only the Reason struct or tags pass information
between BackupOp and kopia

Instead of having a separate struct (OwnersCats) to fetch previous
snapshots, generate and use reasons. While this results in some repeated
data, it cuts down on the number of distinct structs and simplifies some
of the code for getting previous manifests.

A future PR should create a shared function to create a service/cat tag
given a reason.

Only pass in a set of tags to BackupCollections. This pushes the onus
of generating the tags for later snapshot lookups to BackupOp and
creates a somewhat asymmetric interface as Reason is used for the lookup
but tags is used for the backup. This will be updated later so that both
paths use a common function to convert from Reason->tags.

Despite that, it may result in a cleaner interface with kopia (depending
on how far we want to push it) where tags become the main mean of
communication.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1916 

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-10 01:10:19 +00:00
Keepers
f3bf09ba8a
protect against missing backups in manifests (#2063)
## Description

If a prior manifest is missing a backup, it should be handled the same way as when a manifest's backup is missing a details ID: the metadata is skipped, and a full backup is performed.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #2062

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-09 22:01:21 +00:00
Keepers
059b860fde
Invert incremental flags, default to incremental (#2051)
## Description

Sets the default run behavior for exchange to use
incremental backups.  The cli feature flag for enabling exchange incrementals has been swapped for a toggle that disables incrementals, forcing a full backup

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1901

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-08 20:22:56 +00:00
ashmrtn
186569087c
Only make incremental backup if Reason:base is 1:1 (#2050)
## Description

Ensure that each (resource owner, service, category) set of data is only sourced from a single base snapshot when doing an incremental backup. If not, fallback to doing a full backup.

Failure to error out or fallback to a full backup may result in repeated or zombie items in the resulting backup as multiple Point-In-Time backups will be used to source the same data

Incomplete manifests are ignored as they are currently only used for kopia-assisted incrementals, not sourcing items/backup details info when making a delta token-based incremental backup

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #1945 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-06 23:10:55 +00:00
Keepers
70a8f53d65
full backup if prev snapshots have no details (#2049)
## Description

In the event that a backup is completed but the details, somehow, isn't persisted, we want the next backup to do a full, instead of an incremental, backup.  If we don't have this protection the following backups could end up in a bad state.  Future changes will add better resilience so that the fallback isn't needed.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1878

## Test Plan

- [x] 💚 E2E
2023-01-06 22:47:16 +00:00
Keepers
2b45cfa617
remove resource owners from scopes (#1895)
## Description
    
Now that resource owners are identified via
the selector itself, rather than each scope, we
can remove the resource owner data from
scope production and data.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-05 21:29:48 +00:00
Keepers
70d5a5ab56
transition api methods to interfaces (#2010)
## Description

replaces the new api client methods with interfaces, to prepare for testing funcions with mocks instead of integration.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-05 21:03:35 +00:00
Keepers
1f26339813
Match resource owners at top of reduce (#1891)
## Description

Checks for resource owner matches in the top
of the reduce func using the selector owners,
instead of waiting until the path match check.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-05 00:59:16 +00:00
Keepers
0d0a7516f0
use selector owners, not scope owners (#1890)
## Description

Migrates code away from pulling the resource
owner from each scope, and instead usees the
selector as the canon identifier of the resource
owner.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 19:39:25 +00:00
ashmrtn
84db56cc70
Allow incremental backups for multiple data categories if some metadata is missing (#2030)
## Description

Don't return nil if we had an error getting metadata. This gives us the best chance possible of having enough metadata retrieved from the best-effort restore execution flow to actually enable incrementals for some data category.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1777 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-04 19:16:52 +00:00
Keepers
978f9304bf
methodize all api funcs to the Client (#2008)
## Description

In order to use the api layer as an interface, we
need the functions therein to be methods, so
that callers can leverage local interfaces.  This
change introduces the api.Client, and begins
to spread it throughout the exchange package,
largely in place of graph servicers.

No logic changes have occurred here.  The only
modifications are what is required to utilize the
api client.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 17:19:44 +00:00
Keepers
e5edbfd77c
first pass migrate graph api to subfolder (#2003)
## Description

In order to establish a standard api around our
graph client usage, and thus be able to mock
for testing, we need to migrate graph client
usage into another pacakge.  This is the first
step in that process of refactoring.

All changes are code relocation importing and
exporting may change as needed.  No logic
was altered.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 01:35:32 +00:00
Keepers
753ede5a1a
tetst fixes and touchups (#1983)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2023-01-04 00:39:33 +00:00
Keepers
1586ed927b
remove 0-item progress bars (#1986)
## Description

Incremental backups can display progress bars
with 0 items to add or remove.  This can cause
end-user confusion when hte bar completes
with 0 items involved.  We should instead keep
those bars hidden.

Also removes the "heuristics" progress step.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1984

## Test Plan

- [x] 💪 Manual
2023-01-03 23:22:20 +00:00
Keepers
ce1f4406f1
No tester errors 2 (#2005)
## Type of change

- [x] 🐹 Trivial/Minor
2023-01-03 21:35:56 +00:00
Keepers
e4e1b4d4d8
add item mutation tests (#1982)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2023-01-03 19:18:51 +00:00
Keepers
ef866da099
add folder muation tests (#1981)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2023-01-02 19:38:58 +00:00
Keepers
35e0415a75
add move folder delta test (#1979)
## Description

Adds a delta test case for exchange incremental
backups by moving one folder into another one.
Also sets up retrieving container IDs in the test
in preparation for other test control.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2023-01-02 18:00:51 +00:00
Vaibhav Kamra
ad691148fe
Mark folder updated when an updated item is added (#1987)
## Description

If an item is updated, update the folder entries it belongs to. 

Also contains a minor refactor - unexport `FolderEntry` to `folderEntry`

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1812 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-29 23:41:04 +00:00