162 Commits

Author SHA1 Message Date
ashmrtn
f69dfc14e2
Wire kopia exclude list up to BackupOp (#2244)
## Description

Expose the global exclude list mechanism in kopia BackupCollections to other components. Add tests for BackupCollections testing new exclude list functionality

Currently wired to always be passed nil

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2243

merge after:
* #2143 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-27 17:09:40 +00:00
ashmrtn
06efad7028
Bulk up kopia wrapper error checking in BackupOp (#2281)
## Description

Expand error checking for kopia wrapper backup. Kopia may report a number of errors for a snapshot operation but may not give error structs for them. Handle this by checking if there were > 0 errors reported and making our own error struct for this if there are.

This will keep the system from reporting a successful backup when kopia had problems reading or uploading data. Errored files in kopia do not become part of the snapshot

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2280 

## Test Plan

- [ ] 💪 Manual
- [ x  Unit test
- [x] 💚 E2E
2023-01-27 01:38:58 +00:00
Keepers
d529d145cb
scrub pii from observe logs (#2285)
## Description

This is a quick hack to satisfy a primary case of PII scrubbing.  We expect to revisit it in the future.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #2284

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-01-27 00:31:57 +00:00
Vaibhav Kamra
bdb7f2b109
Re-use client for OneDrive item downloads (#2276)
## Description

This PR reintroduces the changes from #2266 with a change to *not* reset the transport
when initializing the shared client. 

Doing so was removing the retry and other middlewares
and also resulting in throttled requests being masked as success

Also - we now decorate our download traffic with an ISV tag as recommended [here](https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic)

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2266 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-01-26 15:05:12 +00:00
Vaibhav Kamra
e9354a8429
Revert "add itemClient for re-usable od item downloads" (#2273)
Reverts alcionai/corso#2266. Am seeing issues on data restore that need to be debugged.
2023-01-26 07:52:56 +00:00
Keepers
bda8a5c60c
add itemClient for re-usable od item downloads (#2266)
## Description

onedrive currently constructs a new http client
for every file it downloads.  This causes the OS
to generate extra sockets, and hang onto them
after the download is complete.  Replacing these
one-off clients with a singular, re-used client-
which is the behavior and standard suggested
for golang http clients- minimizes system
resource consumption.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* closes #2262

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-01-26 04:27:52 +00:00
Keepers
78b9f2752e
adds clues package to begin structured logging (#2214)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1969

## Test Plan

- [x] 💪 Manual
2023-01-25 21:39:38 +00:00
Keepers
e828209a30
return gc stats errors in backup (#2248)
## Description

If gc.stats reports a non-zero error count at the
end of a backup, retrieve the error from the
status and return it as the backup operation err.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-25 10:44:00 +00:00
Vaibhav Kamra
4a6959f60f
Revert "Issue 2022 eventincr" (#2256)
Reverts alcionai/corso#2167. This is causing the integration test to fail consistently (in CI and local runs)
2023-01-25 10:21:00 +00:00
Keepers
39c3c8a895
Issue 2022 eventincr (#2167)
## Description

Fixes and unskips the tests incremental integration tests
for exchange calendar events.

Currently blocked on LynneR displaying the calendar
fetch issues where it's only possible to retrieve her default
Calendar, Birthdays, and US Holidays calendars.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #2022

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-01-23 21:38:09 +00:00
ashmrtn
7f44db6c7c
Fail operation if we failed to fetch item data (#2199)
## Description

If GraphConnector reported errors while trying to fetch item data, fail the entire operation. This stops silent failures, but is a big hammer at the moment because there's no checks for items no longer being available.

Error reporting is minimal as well, but some info should be in the log.

**This effectively disables best-effort as any failure will result in the operation being reported as failed**

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2196

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-20 18:59:15 +00:00
Keepers
51e29f2975
use delta queries for calendar events (#2154)
## Description

Hacks the beta version call into the events api
when iterating through items, allowing us to run
delta-based queries for calendar events.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #2022

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-19 01:30:56 +00:00
ashmrtn
f01c8ad843
Add logging for bases in incremental backups (#2151)
## Description

Add log statements noting which bases were used for kopia assisted incrementals and which bases were merged into the hierarchy. Also record the reasons a base was chosen.

Log statements when searching for previous snapshots will be added when that code is refactored

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #2149 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-17 21:51:57 +00:00
Keepers
48e4b65165
migrate, test manifest+meta producer (#2091)
## Description

Adds mocked unit tests for produceManifestsAnd-
Metadata.  For cleanliness, moves that func, and
any funcs called within it, to their own file within operations

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #2062

## Test Plan

- [x]  Unit test
2023-01-17 21:22:30 +00:00
ashmrtn
8d798d6024
Switch to "delete wins" merge strategy for Exchange IDs (#2142)
## Description

#2116 implemented a "last change wins" strategy for deduping IDs. However, it appears Graph API doesn't really enforce an ordering for returned items so that strategy may not work. Instead, implement a "delete wins" strategy.

Items that are added, deleted, and then restored in M365 will be restored with a different ID

This PR does the following:
* reverts changes to return a unified list of IDs and added/removed status
* switches to a "delete wins" merge strategy for duplicate IDs
* updates tests

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #1954 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-13 20:11:25 +00:00
Keepers
5980c4dca0
log observed messages (#2073)
## Description

add logging to the observe package, assume that
every instance where a message is observed, it
also gets logged.

Merger may want to wait until logging to a file is the standard behavior, else the terminal might get messy/buggy.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #2061

## Test Plan

- [x] 💪 Manual
2023-01-13 19:34:01 +00:00
ashmrtn
f43c458844
Dedupe collection item IDs prior to fetching item data (#2116)
## Description

Under some conditions, items can be returned multiple times when enumerating the items in a folder in Exchange. Begin deduping those items by doing the following:
* use a map in ExchangeDataCollection to dedupe on item IDs
* have enumeration function return a single slice of items containing both additions and deletion so a "last-change-wins" policy can be implemented

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1954 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-13 02:19:21 +00:00
Vaibhav Kamra
cfe6549987
Re-Enable Kopia-assisted incrementals for OneDrive (#2126)
## Description

This addresses the deadlock in the item progress reader by deferring the reader creation to
when the first read is issued for the item

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1702 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-12 03:50:26 +00:00
Keepers
580934b069
remove DiscreteScopes() from selectors (#2036)
## Description

DiscreteScopes is a vestigial func from when scopes contained the list of resource owners to track.  That behavior is no longer in use.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
2023-01-10 23:24:01 +00:00
ashmrtn
e88553d073
Refactor the Reason struct and how tags for lookups are generated (#2087)
## Description

Solidify the interface between BackupOp and KopiaWrapper by making Reason the de facto way to pass/generate tags for things. The Reason struct now includes a function to generate tags for that instance. KopiaWrapper also now hides the fact that it prefixes tags from other components

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1916 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-01-10 20:32:17 +00:00
ashmrtn
43c2017492
Don't use OwnersCats struct (#2055)
## Description

Remove OwnersCats so only the Reason struct or tags pass information
between BackupOp and kopia

Instead of having a separate struct (OwnersCats) to fetch previous
snapshots, generate and use reasons. While this results in some repeated
data, it cuts down on the number of distinct structs and simplifies some
of the code for getting previous manifests.

A future PR should create a shared function to create a service/cat tag
given a reason.

Only pass in a set of tags to BackupCollections. This pushes the onus
of generating the tags for later snapshot lookups to BackupOp and
creates a somewhat asymmetric interface as Reason is used for the lookup
but tags is used for the backup. This will be updated later so that both
paths use a common function to convert from Reason->tags.

Despite that, it may result in a cleaner interface with kopia (depending
on how far we want to push it) where tags become the main mean of
communication.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1916 

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-10 01:10:19 +00:00
Keepers
f3bf09ba8a
protect against missing backups in manifests (#2063)
## Description

If a prior manifest is missing a backup, it should be handled the same way as when a manifest's backup is missing a details ID: the metadata is skipped, and a full backup is performed.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #2062

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-09 22:01:21 +00:00
Keepers
059b860fde
Invert incremental flags, default to incremental (#2051)
## Description

Sets the default run behavior for exchange to use
incremental backups.  The cli feature flag for enabling exchange incrementals has been swapped for a toggle that disables incrementals, forcing a full backup

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1901

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-08 20:22:56 +00:00
ashmrtn
186569087c
Only make incremental backup if Reason:base is 1:1 (#2050)
## Description

Ensure that each (resource owner, service, category) set of data is only sourced from a single base snapshot when doing an incremental backup. If not, fallback to doing a full backup.

Failure to error out or fallback to a full backup may result in repeated or zombie items in the resulting backup as multiple Point-In-Time backups will be used to source the same data

Incomplete manifests are ignored as they are currently only used for kopia-assisted incrementals, not sourcing items/backup details info when making a delta token-based incremental backup

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* closes #1945 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-01-06 23:10:55 +00:00
Keepers
70a8f53d65
full backup if prev snapshots have no details (#2049)
## Description

In the event that a backup is completed but the details, somehow, isn't persisted, we want the next backup to do a full, instead of an incremental, backup.  If we don't have this protection the following backups could end up in a bad state.  Future changes will add better resilience so that the fallback isn't needed.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1878

## Test Plan

- [x] 💚 E2E
2023-01-06 22:47:16 +00:00
Keepers
2b45cfa617
remove resource owners from scopes (#1895)
## Description
    
Now that resource owners are identified via
the selector itself, rather than each scope, we
can remove the resource owner data from
scope production and data.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-05 21:29:48 +00:00
Keepers
70d5a5ab56
transition api methods to interfaces (#2010)
## Description

replaces the new api client methods with interfaces, to prepare for testing funcions with mocks instead of integration.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-05 21:03:35 +00:00
Keepers
1f26339813
Match resource owners at top of reduce (#1891)
## Description

Checks for resource owner matches in the top
of the reduce func using the selector owners,
instead of waiting until the path match check.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-05 00:59:16 +00:00
Keepers
0d0a7516f0
use selector owners, not scope owners (#1890)
## Description

Migrates code away from pulling the resource
owner from each scope, and instead usees the
selector as the canon identifier of the resource
owner.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1617

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 19:39:25 +00:00
ashmrtn
84db56cc70
Allow incremental backups for multiple data categories if some metadata is missing (#2030)
## Description

Don't return nil if we had an error getting metadata. This gives us the best chance possible of having enough metadata retrieved from the best-effort restore execution flow to actually enable incrementals for some data category.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1777 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-01-04 19:16:52 +00:00
Keepers
978f9304bf
methodize all api funcs to the Client (#2008)
## Description

In order to use the api layer as an interface, we
need the functions therein to be methods, so
that callers can leverage local interfaces.  This
change introduces the api.Client, and begins
to spread it throughout the exchange package,
largely in place of graph servicers.

No logic changes have occurred here.  The only
modifications are what is required to utilize the
api client.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 17:19:44 +00:00
Keepers
e5edbfd77c
first pass migrate graph api to subfolder (#2003)
## Description

In order to establish a standard api around our
graph client usage, and thus be able to mock
for testing, we need to migrate graph client
usage into another pacakge.  This is the first
step in that process of refactoring.

All changes are code relocation importing and
exporting may change as needed.  No logic
was altered.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1967

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-01-04 01:35:32 +00:00
Keepers
753ede5a1a
tetst fixes and touchups (#1983)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2023-01-04 00:39:33 +00:00
Keepers
1586ed927b
remove 0-item progress bars (#1986)
## Description

Incremental backups can display progress bars
with 0 items to add or remove.  This can cause
end-user confusion when hte bar completes
with 0 items involved.  We should instead keep
those bars hidden.

Also removes the "heuristics" progress step.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1984

## Test Plan

- [x] 💪 Manual
2023-01-03 23:22:20 +00:00
Keepers
ce1f4406f1
No tester errors 2 (#2005)
## Type of change

- [x] 🐹 Trivial/Minor
2023-01-03 21:35:56 +00:00
Keepers
e4e1b4d4d8
add item mutation tests (#1982)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2023-01-03 19:18:51 +00:00
Keepers
ef866da099
add folder muation tests (#1981)
## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2023-01-02 19:38:58 +00:00
Keepers
35e0415a75
add move folder delta test (#1979)
## Description

Adds a delta test case for exchange incremental
backups by moving one folder into another one.
Also sets up retrieving container IDs in the test
in preparation for other test control.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2023-01-02 18:00:51 +00:00
Vaibhav Kamra
ad691148fe
Mark folder updated when an updated item is added (#1987)
## Description

If an item is updated, update the folder entries it belongs to. 

Also contains a minor refactor - unexport `FolderEntry` to `folderEntry`

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1812 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-29 23:41:04 +00:00
Keepers
6419a59636
set up integration test framework (#1975)
## Description

Adds the base test framework for handling
incremental backups integration testing.  Future
changes will add more tests to the table.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2022-12-28 23:31:35 +00:00
Keepers
673ee10752
add happy-path incremental tests to operations (#1973)
## Description

Add a simple happy-path integration test to
operations backups.  This test only attemps to
assert the most basic expectations: that
incrementals are runnable, and that they involve
less data than the initial backup.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2022-12-28 21:13:20 +00:00
Keepers
0e6fce7d1e
another cleanup refinement (#1971)
## Description

A little code consolidation in ops/backup integration
tests, to prepare testing incrementals.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2022-12-28 20:23:07 +00:00
Keepers
fe23d81c0a
split op/backup tests into two files (#1968)
## Description

A quick refactor to clean up the code and split
op/backup testing between unit and integration
test sets, so that we don't bloat a single file.

No actual changes, just lift and shift.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🤖 Test

## Issue(s)

* #1966

## Test Plan

- [x] 💚 E2E
2022-12-28 16:58:52 +00:00
Keepers
9e9cb43b58
retain kopia-assist when not on incrementals (#1941)
## Description

Adds a flag to the BackupCollections interface
that identifies whether the caller is running an
incremental backup or not.  If they are, kopia
will utilize the previous base snapshots when
building the directory tree.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1901

## Test Plan

- [x] 💚 E2E
2022-12-23 20:14:05 +00:00
ashmrtn
35c1b10912
Wire details merging during BackupOp.Run (#1905)
## Description

Actually call the details merge function while performing a backup. If no base snapshots or `len(toMerge) == 0` then turns into a noop.

This PR contains a little extra refactoring to keep from initializing a struct multiple times. However, long-term we should probably refactor BackupOp to better fit the fact that we are now using interfaces as inputs to many functions for the sake of testing

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1800 

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-12-23 19:14:31 +00:00
ashmrtn
62c8f8c6ea
Update details merge function for details Builder and new location of folder population (#1926)
## Description

Folder population now done when merging items in BackupOp. Also add more tests to make sure folders are actually populated properly in Details

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-23 02:04:14 +00:00
ashmrtn
bc56d38970
Functions to merge a set of backup details (#1904)
## Description

Go through the provided bases, load their backup details, and check if
any of the items in them need to be merged into the details for the
current backup.

Has a small amount of logic to treat moved items as updated.

Also include all the tests

Viewing by commit may be useful

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-23 01:18:08 +00:00
ashmrtn
6b689b76c1
Produce map of items that need backup details entries (#1892)
## Description

Generate a `map[old ShortRef]new path.Path` for every item sourced from a base snapshot during backup. Return this information at the end of BackupCollections so that callers can use it to merge backup details if desired.

This PR does a few auxiliary things as well:
* thread current and previous path through hierarchy merging
* add extra field to info tracked by corsoProgress to hold previous path
* provide current and previous paths to streamBaseItems function

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-12-22 22:29:15 +00:00
ashmrtn
f37ee2b942
Factor out function to retrieve backup details (#1894)
## Description

Given a backup ID pull the backup details for the backup. This function will also be used during incremental backups as the backup details from the base snapshot(s) will need merged with the details generated by backing up new/changed items.

## Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #1800 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-12-22 20:35:10 +00:00
Keepers
5b6b60de60
enable incrementals with feature flags (#1903)
## Description

Allows the usage of incrementals via feature-
flag controls.  Feature flags can be enabled
on a per-flag basis in either the cli (the flag
is hidden from users, normally) or through the
sdk by directly toggling the flag property
in the control.Options.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1901

## Test Plan

- [x] 💪 Manual
2022-12-21 21:31:48 +00:00