1072 Commits

Author SHA1 Message Date
Keepers
fe148af455
normalize bucket naming during config lookup (#1025)
## Description

Ensure s3 bucket name inputs are normalized
in all uses.  Currently, the normalized format is
applied when using storage, but not when passing
around --bucket flag values elsewhere.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #975

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-10-03 21:37:20 +00:00
Danny
d60a771024
GC: TWO Message attachments (#1014)
## Description
Introduces mock testing to ensure that objects with two or more attachments are restored properly.

Minor change to restore pipeline for failure. attachments are restored separately in the event that one of the attachments is corrupted. 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1007<issue>

## Test Plan

- [x]  Unit test
2022-10-03 20:59:03 +00:00
ashmrtn
56287c42fa
Skip kopia upload when no collections given (#846)
## Description

<!-- Insert PR description-->
Currently does not throw an error. Open to other suggestions on return values.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #844 
* #1000 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-03 17:49:20 +00:00
Keepers
7fa86824e9
Issue 1020 (#1021)
## Description

CI tests are showing a consistent panic when adding attendees
to events with zero content.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #1020

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-10-03 17:17:32 +00:00
Vaibhav Kamra
03bb63f52d
Use selectors in OneDrive CLI (#996)
## Description

Adds the following selectors to OneDrive details/restore :
- `file-name`, `folder`, `file-created-after`, `file-created-before`, `file-modified-after`, `file-modified-before`

Also includes a change where we remove the `drive/<driveID>/root:` prefix from parent path entries in details. This
is to improve readability. We will add drive back as a separate item in details if needed later.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #627 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-03 07:23:30 +00:00
ashmrtn
88af7f9b7c
restore/backup tests for events (#995)
## Description

Test restoring a single calendar and multiple calendars of events

Manually compare Location models because they have some fields that are marked as for internal use and appear to be overwritten/controlled by M365

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-30 23:39:33 +00:00
ashmrtn
526cc8fa9b
Add code to disable default scheduling policy (#1010)
## Description

All backups with corso are manual and use data sourced outside of kopia's control. Disable kopia's scheduling policy so we don't accidentally confuse it

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1009

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-30 23:10:10 +00:00
ashmrtn
b612718723
Update time parser to handle OneDrive format (#1013)
## Description

OneDrive format uses '-' in place of ':'. Purge command needs to be able to clear folders with this format as well.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1012

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-30 22:38:43 +00:00
Danny
a31a17ce1f
GC: Extend mockable to include attachments for mail (#990)
## Description
Mockable `exchange.Mail.Messages` extended to include direct attachments and oneDrive attachments. 

NOTE: There are changes to support as it was recognized that the lack of these variables during cloning caused Issue #978 for direct attachments. However, it does not solve it for OneDrive. 

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix


## Issue(s)


**closes** #988<issue>

## Test Plan

- [x]  Unit test
2022-09-30 14:08:17 +00:00
ashmrtn
5da8410ebf
Test backing up multiple top-level folders in mail (#993)
## Description

Use multiple restores to populate the account with known data and then run a backup over the restores to pull the data.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-30 01:00:10 +00:00
ashmrtn
0cba7a402c
Restore tests for contacts (#989)
## Description

Test restoring a single contacts folder and multiple contacts folders. This tests that items from different folders can be restored, but not that items from multiple folders can be backed up (because the restore colocates into a single folder)

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

merge after
* #986 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-30 00:32:28 +00:00
ashmrtn
1a188ce657
Extra checks for kopia retention policy (#964)
## Description

Add a regression test component to the retention test. This ensures that setting all values to 0 causes it to keep MaxInt latest snapshots.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #906

pending merge of
* kopia/kopia#2454

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-29 23:55:35 +00:00
ashmrtn
4e86cd2f84
Fix bug fetching contact folders with filters (#986)
## Description

Stop fetching all folders for every new folder seen, erasing previous results. Regression test to come in a later patch as it requires some extra infrastructure

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #982 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-29 23:20:50 +00:00
Danny
114fec6059
GC: Restore: Simplified Event Restore (#981)
## Description
Implementation of simplified Restore based on the following [spec](https://www.notion.so/alcion/Event-restore-semantics-061aee5288244629b1c53337e4dea306#6e974540c8804c4fa832218675534e1c)

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #954<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2022-09-29 22:01:12 +00:00
ashmrtn
23e1db13df
Minor revisions to comparisons in restore/backup tests (#987)
## Description

* treat empty values and nil pointers as equal when comparing Graph models
* make sure all items are always iterated through so the test doesn't hang
* explicitly check number of items retrieved
* remove some unneeded casts and assignments

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-29 17:26:13 +00:00
Keepers
d4390ac5ea
add diagnostics to load_test (#983)
## Description

Adds some go runtime diagnostics tracking to
load testing, including some trace regioning.
Unfortunately, I couldn't find any third party trace library that didn't depend on a sidecar server
to sample against the application on.  Therefore,
just starting with something basic.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-09-29 16:47:09 +00:00
ashmrtn
f0ff7ee982
Basic checks for bad/empty values for restore (#973)
## Description

Test bad services or empty collections don't cause an error

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-28 22:41:50 +00:00
ashmrtn
3c0179986b
Restore and backup tests for GraphConnector (#980)
## Description

Add framework and tests for some exchange mail restore and backup situations. Framework can be used to test other situations in the future, this is just the starting point.

Some logic in the test can be further generalized/factored out once we know more about how paths will be transformed during restore

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-28 20:23:54 +00:00
Danny
652bb9cd4a
GC: Restore: Contact: Restore to single Folder (#972)
## Description

The feature ensures that this Flat style directory only creates a single folder and restores all the contacts to the same folder during the restore process. 

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #880<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-28 14:16:08 +00:00
ashmrtn
507eab088d
Implement Size() in mock connector (#974)
## Description

Adds the ability to use mockconnector for onedrive and exchange restores, reducing potential code duplication overall.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-28 01:38:10 +00:00
ashmrtn
0a629e0807
Wire up RestoreDestination (#949)
## Description

* No new functionality exposed to CLI users
* generate restore folder names in CLI and pass down the stack
* update tests for new parameters
* centralize generation of restore container names

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #897 
* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-27 16:06:15 +00:00
Keepers
ed2dd9a026
scope integration tests to specific folders (#806)
## Description

Exchange app e2e tests need to scope their
backup dataset to only the primary folder, else
we risk exponential duplication of data when
the next backup picks up all the corso-copied
folders.

## Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix

## Issue(s)

* #501
### Awaiting completion 
- #810 
- #811
## Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-09-27 14:09:10 +00:00
Danny
b57829cc05
MockData: Mail Updated w/ Current Escape Policy (#959)
Mockdata exchanged for a newer version Kiota serialized version.

## Description

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #956<issue>

## Test Plan

- [x]  Unit test
2022-09-27 01:20:05 +00:00
Vaibhav Kamra
7d1cf2ce5b
Use path based addressing to get OneDrive folder (#951)
## Description

Addresses a follow-up from the folder lookup implementation where we were not iterating through
all children (no pagination was implemented) when searching for a folder.

Instead of implementing pagination, this uses OneDrive path based addressing to do a direct lookup
for the folder.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #875 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-26 23:53:22 +00:00
Keepers
984df8fcfb
add bup/restore ids for events (#946)
## Description

Start-end event pairs want ids as a means to
match each pair.  Restore operations now gen
a transient id for this purpose.  Backup ops pre-
generate the base ID instead of waiting for the
model store to generate the ID at the end of
the operation.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #894

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-26 23:22:49 +00:00
Vaibhav Kamra
1df48997ae
Add additional OneDrive metadata to backup details (#952)
## Description

Adds file size and timestamps to enable selectors using this metadata

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #594 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-26 22:54:42 +00:00
Vaibhav Kamra
ea73873ffb
Fix OneDrive restore regressions (#953)
## Description

Fixes a couple of regressions that got introduced on the restore path via code review changes and
kopia wrapper refactor.

Found while writing the OneDrive restore test. Would have been prevented if we had said test.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-26 22:20:54 +00:00
Keepers
38addfaae8
track bytes written, read in kopia (#944)
## Description

Track the count of bytes read and written in
kopia.  For backups, this means the count of
bytes fed into kopia (hashed bytes), and the
amount written after compression and dedupe
(total file bytes).  For restore, this is the count of bytes in all files read.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #894

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-26 19:18:42 +00:00
ashmrtn
ff2db0c553
Export path package to other codebases (#912)
## Description

Moves the `path` package to the `pkg` package so other code outside of Corso can use it if they need it

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #908 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-26 18:34:19 +00:00
Danny
06d2a389d9
GC: Backup: Events: Filter Calendars with Scope (#900)
## Description
Feature branch enabling filtering based upon `selectors.ExchangeScope`
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #810<issue>

## Test Plan
- [x]  Unit test
2022-09-26 17:33:33 +00:00
Keepers
679b32697b
track backup creation time on restore start (#918)
## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #894

## Test Plan

- [x] 💪 Manual
2022-09-23 20:22:06 +00:00
Danny
f39a14eda3
GC: const variables extended to include default container names (#886)
## Description
Moves constants to a file and adds more source definitions and backgrounds to constants.
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #823<issue>

## Test Plan
Const variables to be used in future PRs. No tests are available. 
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-23 14:17:59 +00:00
Danny
3f3c8217a3
GC: Component: Version 40.0 (#917)
## Description
Upgrades Graph API to the newest version. Breaking change from the previous version, all functionality needs to be inspected to ensure operation.  
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #828<issue>

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-22 18:59:51 +00:00
Keepers
76f2c5a632
improve language on selector reduce error (#794)
## Description

shorten and consolidate the error message when
the provided selectors match zero entries in the
backup entity list.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #564

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-22 15:44:08 +00:00
Keepers
d633be9eef
nightly load test footprint and workflow (#928)
## Description

Adds the footprint file for running nightly load
tests out of the repository package.  The tests
don't do anything at this stage, but that's okay.
Also introduces a workflow to kick off the tests
every day at 2am UTC.

## Type of change

- [X] 🤖 Test
- [X] 💻 CI/Deployment

## Issue(s)

* #902

## Test Plan

- [X] 💚 E2E
2022-09-22 15:16:25 +00:00
Keepers
ce4d97ef48
report totalFileSize on backup (#921)
## Description

Extend the kopia metrics to include the snapshot's TotalFileSize parameter.  This value also gets used as the metric for BytesWritten at the end of a
backup.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #894

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-21 17:43:16 +00:00
ashmrtn
975b8b8e5b
Bulk up tests for kopia package some (#920)
## Description

Largest changes around
* Adding tests for some init functions for conn
* Checking modelstore get metadata with tags functionality
* wrapper is best-effort for restore when there's errors

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-21 16:13:27 +00:00
Keepers
c3176cf113
Count number of resources in a slice of colls (#916)
## Description

Backup and Restore metrics need to know the
count of resource owners in the operation.  This
processor extracts that data from the collections
that get passed from producer to consumer.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #894

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-20 23:09:26 +00:00
ashmrtn
7850272a5e
Apply wsl linter to onedrive package (#911)
## Description

Finish enabling wsl linter on Corso repo by removing the exception for onedrive package

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #481

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-20 20:02:35 +00:00
ashmrtn
acf1837e77
909 path folders (#910)
## Description

Adds function call to the path package that returns the folder elements present in a path after the path prefix. This is safer than using `strings.Split()` because some paths may have `/` characters in elements.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #909 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-20 17:15:46 +00:00
ashmrtn
8678c0b159
Disable kopia retention policy (#904)
## Description

Disable the kopia retention policy that it uses by default. All snapshot deletions in corso must be done manually.

There should really be a follow up change to this after making a method in kopia public. Right now this PR is going off [some code](7bda16ab33/snapshot/policy/retention_policy.go (L117)) that shows the effective number of "latest" snapshots will default to intMAX if all of the other fields are set. We really should have a regression test or something for this to ensure that's actually the case. I've manually tested this patch, but that's not a scalable solution

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #889 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-20 15:09:05 +00:00
Vaibhav Kamra
921d77cc05
Implement OneDrive DataCollection restore (#877)
## Description

Implements `data.Collection` restore for OneDrive and wires it up to the restore operation

Leverages previously added `onedrive` helpers that have integration tests.

An op/CLI level integration test will be added in a follow-up PR.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #668 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E

```
$ ./corso restore onedrive --backup 7de4d68f-67c2-4e81-95a2-81cf0d1bd11d
Restored OneDrive in S3 for user
```
2022-09-19 21:46:30 +00:00
Keepers
1203fd2b6a
extract timestamp from arbitrary string (#883)
## Description

In order for purge to gracefully handle the variety of timestring values that may appear within a
folder name, the common/time.go file has added
a func to extract a time substring from a string.
Purge is switched to that extractor instead of
a len-based suffix check.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #805

## Test Plan

- [x] 💪 Manual
2022-09-19 20:35:49 +00:00
Keepers
a0508cc442
use account id, mock event bus (#870)
## Description

1/ introduce an ID() method int the account
package which provides the tenantID of the
configured account provider.

2/ introduce a event bus mock for testing.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #741

## Test Plan

- [x]  Unit test
2022-09-19 19:40:43 +00:00
Danny
f250665dd6
GC: Backup: Contacts: Ability to filter on folder [FEATURE] (#891)
## Description
PR enables Filtering for `exchange.Contact` items wherein if an `exchange.ContactFolder` is requested specifically, only the requested folders are backed up. 
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes  #811<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-19 17:42:12 +00:00
ashmrtn
423b6e19f7
Encode paths given to kopia (#892)
## Description

Encode all paths given to kopia with base64URL to ensure no special characters that kopia can't handle end up in there. The encoded paths are not stored in backup details nor are they ever surfaced to the user. This also works around the previous limitation where Corso was unable to properly backup or restore exchange email layouts that had folders containing `/` characters

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #865

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-19 16:49:33 +00:00
Danny
14c437b48f
GC: Iteration Cleanup (#878)
## Description
Small refactor. Moves Path functions from iterator to `service_functions.go`
Introduces getCollectionPath which combines the two previous functions and reduces code complexity
## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

- [x]  Unit test
2022-09-16 22:30:36 +00:00
ashmrtn
09343c0220
Coalesce items into collections in kopia/wrapper.go during restore (#888)
## Description

The wrapper.go file was not properly grouping items during restore, instead returning a bunch of single item data.Collections. Update this to properly group items by folder. Also update the tests to be more strict about checking for this.

Remove dead code around restoring a single item as it was unused. Lose some visibility into what happens if the code is unable to open the reader for a given item.

Generally moves some logic around in wrapper.go

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #879 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-16 22:12:30 +00:00
Keepers
609bbc3512
skip tests that restore contacts, events (#885)
## Description

Skips any integration tests that restore events or contacts.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #884

## Test Plan

- [x] 💚 E2E
2022-09-16 21:33:57 +00:00
ashmrtn
275961b5bd
Add more data stored in the test suite (#887)
## Description

No change to logic in other code, just refactoring the tests

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-16 21:18:43 +00:00