371 Commits

Author SHA1 Message Date
Keepers
a32f86e40d
migrate naming of azure env vars (#1176)
## Description

migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.

⚠️  Your local environment will need to change in the following ways:  ⚠️ 
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #558

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-10-17 20:47:04 +00:00
Eno Thereska
fb46f5c7f3
Export call to get user ids (#1069)
## Description

Exposing user IDs in addition to user emails.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [X] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [X]  Unit test
- [ ] 💚 E2E
2022-10-17 18:17:39 +00:00
ashmrtn
72a3c7ab3b
Trim trailing '/' from CLI input for restore/details (#1175)
## Description

Use paths package to trim unescaped trailing '/' characters from input for Exchange mail and OneDrive folder names. Add tests for Exchange showing that the trimming also works properly if the folder name ends with '/'.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1147 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-17 17:54:36 +00:00
Keepers
d2d5cd7f42
load test all data for all users (#1172)
## Description

Watch the minimum-complete load test over the weekend.
Backup only, all tenant users, all data types, no restrictions.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💚 E2E
2022-10-14 19:11:46 +00:00
Keepers
7facdf1cc3
Issue 902 onedriveindividual (#1140)
## Description

Added an additional set of test suites to load_test
to focus on the backup of a single user who will
have their account prepared with a significant amount
of data (many emails, many events, large files, etc).

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902 

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-14 17:45:06 +00:00
Keepers
d4bf3f872f
scrutinize load test results for all users (#1118)
## Description

Adds a check that reviews the load test details.
Details entries to make sure that all users were
involved in the operation, and that for each user,
the test involved all item categories for that app.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-14 00:22:38 +00:00
Keepers
e5243404a8
add prefix option to scopes (#1141)
## Description

adds extensible options to folder-level scopes that allows the caller to specify whether they want a
prefix-comparison matcher or a contains-comparison matcher.

Also corrects the behavior of the prefix filter so that it accurately follows the "target is prefix of input" specification, rather than the reverse.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #1133

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-10-13 22:58:18 +00:00
Georgi Matev
0c921e8b40
Normalize repo suffix paths (#1153)
## Description

Repo `--prefix` values should be normalized with a trailing `/`

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* Fixes #1152  

## Test Plan

<!-- How will this be tested prior to merging.-->
- [] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-13 02:11:13 +00:00
ashmrtn
ec916bc1b3
Test ensuring shortrefs are unique for edgecase (#1105)
## Description

Some paths can generate identical unescaped strings even though they have different elements. Add a test ensuring they generate unique ShortRefs even under these conditions.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1092 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-10 22:47:23 +00:00
Keepers
0b65b20200
attach logger to test ctx, use flusher (#1083)
## Description

Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context.  The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton.  But in order for the logs to show up
within the test, they need to get flushed.

See my comments for focus on non-chaff changes.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-10-10 21:14:10 +00:00
Keepers
89668ed164
describe time value standards, validate time flags (#1066)
## Description

Adds validation of cli inputs for time-based flags.
All time values that can be parsed by the time
handling in common count as supported, even
though official support includes a smaller set.

Also adds some clean-up to time.go and adds a
design document describing standard time value
formats, and the maintenance thereof, in corso.

## Type of change

- [x] 🌻 Feature
- [x] 🗺️ Documentation
- [x] 🤖 Test

## Issue(s)

* #943
* #924

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-10-06 17:46:21 +00:00
Keepers
cba065ec94
reduce load test to default folders (#1062)
## Description

Load Testing is causing folder growth (fix is underway)
and is already taking > 4 hours only to back-up emails.
This reduces the dataset to only the primary folders for
each exchange type.  We can expand from here when
data production is better under control.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💚 E2E
2022-10-05 23:42:04 +00:00
ashmrtn
3f0951dea2
Higher resolution on test restore folder names (#1040)
## Description

\~Only affects folders created by tests\~
Create and use helper function that sets the restore destination folders to have microsecond granularity. This is to avoid naming collisions with second granularity (of which I've observed at least once)

Switches all tests for all services/categories to use OneDrive DateTime format+microseconds

Manually verified that a manually created folder with the given timestamp format was removed by the CI purge script

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* closes #1036

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-04 23:52:13 +00:00
Keepers
3719c36bce
print restore details following successful op (#1030)
## Description

Now that GC is tracking the details entries during
a restore procedure, it can display the results in
the cli.  Due to an absence of itemInfo, only the
item shortRef is displayed.  But we can expand
on that from here.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #977

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-10-04 20:43:38 +00:00
Vaibhav Kamra
3df3d68a3e
OneDrive details UX improvements (#1034)
## Description

- Rename `LastModified` to `Modified` in OneDrive backup details
- Humanize file size units

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #627 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-04 15:44:47 +00:00
Keepers
fe148af455
normalize bucket naming during config lookup (#1025)
## Description

Ensure s3 bucket name inputs are normalized
in all uses.  Currently, the normalized format is
applied when using storage, but not when passing
around --bucket flag values elsewhere.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #975

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-10-03 21:37:20 +00:00
Keepers
11be4e4590
up the frequency of load tests (#1024)
## Description

Temporarily up the frequency of the load test kickoff for observation.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💚 E2E
2022-10-03 18:59:36 +00:00
Vaibhav Kamra
03bb63f52d
Use selectors in OneDrive CLI (#996)
## Description

Adds the following selectors to OneDrive details/restore :
- `file-name`, `folder`, `file-created-after`, `file-created-before`, `file-modified-after`, `file-modified-before`

Also includes a change where we remove the `drive/<driveID>/root:` prefix from parent path entries in details. This
is to improve readability. We will add drive back as a separate item in details if needed later.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #627 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-10-03 07:23:30 +00:00
ashmrtn
dd7b84871f
CLI and selectors tests for folder prefix match (#1011)
## Description

Specifying a folder on the CLI should cause the system to do a prefix match on that folder when doing a restore, backup details, or selector reduce operation.

Recently updated behavior, adjusting tests to match new expectations.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-30 20:52:19 +00:00
Keepers
d4390ac5ea
add diagnostics to load_test (#983)
## Description

Adds some go runtime diagnostics tracking to
load testing, including some trace regioning.
Unfortunately, I couldn't find any third party trace library that didn't depend on a sidecar server
to sample against the application on.  Therefore,
just starting with something basic.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2022-09-29 16:47:09 +00:00
Vaibhav Kamra
b7f5ed73c3
OneDrive Folder and Item selectors (#985)
## Description

Adds support for folder and item selection

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #627 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-29 02:44:08 +00:00
Keepers
011e24583f
add profiling, tracing to load testing (#971)
## Description

add diagnostic records to load test runs

## Type of change

- [x] 🤖 Test
- [x] 💻 CI/Deployment

## Issue(s)

* #902

## Test Plan

- [x] 💚 E2E
2022-09-29 02:01:15 +00:00
Vaibhav Kamra
e59d596152
Fix selector path matching (#979)
## Description

Fixes an issue with the path matching logic in selectors where if the path was a substring of what was
specified as the scope in the filter, the item would still be matched.

e.g. 
Given 2 items - `/fold/contact1` and `/folderA/folderB/contact2` and a 
selector `er.Include(er.ContactFolders("AnyUser", []string{"folderA/folderB"}))`, 
the selector would match both items because `fold` is contained in the selector scope `folderA/folderB`

The fix is to invert the comparison - we check if the selector scope is contained in the item path instead.
In the example above, the selector string `folderA/folderB` will then only match the second items path.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor


## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-29 00:06:26 +00:00
ashmrtn
0a629e0807
Wire up RestoreDestination (#949)
## Description

* No new functionality exposed to CLI users
* generate restore folder names in CLI and pass down the stack
* update tests for new parameters
* centralize generation of restore container names

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #897 
* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-27 16:06:15 +00:00
Keepers
e061ce7c73
implement load tests (#937)
## Description

Adds the load test func calls and asserts for
both exchange and onedrive.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💚 E2E
2022-09-27 00:21:30 +00:00
Keepers
984df8fcfb
add bup/restore ids for events (#946)
## Description

Start-end event pairs want ids as a means to
match each pair.  Restore operations now gen
a transient id for this purpose.  Backup ops pre-
generate the base ID instead of waiting for the
model store to generate the ID at the end of
the operation.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #894

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-26 23:22:49 +00:00
Vaibhav Kamra
1df48997ae
Add additional OneDrive metadata to backup details (#952)
## Description

Adds file size and timestamps to enable selectors using this metadata

## Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #594 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-26 22:54:42 +00:00
ashmrtn
7b5c8d43f1
Create POD struct with restore destination info (#947)
## Description

This PR has just the struct definition to allow conversation about the design if needed. Future PR(s) will create and populate the struct during restore (in the CLI-layer since this struct is meant to be used to communicate that information) and pass it down the stack to the graph connector

The roadmap has plans for allowing users to

1. change the "root" folder the restore occurs in
1. change the resource owner of the restored items (e.x. restoring items from one user to a different user)

Both options can be enabled at the same time (e.x. restore "Inbox" from user A to "UserAInboxRestore" in user B). When the folder is not given it should default to "Corso_Restore_\<date time\>" and when the resource owner is not given the item should be restored under the original resource owner. As of now, we don't expect to have to have fine-grained control over restoring multiple users to different folders on a per-user basis. However, we may need to restore to specify a folder that is created for multiple users and restored to.

Examples:
**Not expected:** Restore UserA Inbox to folder1 in their hierarchy and UserB to folder2 in their hierarchy

**Expected:** Restore UserA Inbox to Corso_Restore_2022-09-23
POD struct:
```go
dest := selectors.RestoreDestination{
  ContainerName: "Corso_Restore_2022-09-23",
}
```

**Expected:** Restore UserA Inbox to UserB Corso_Restore_2022-09-23
POD struct:
```go
dest := selectors.RestoreDestination{
  ResourceOwnerOverride: "UserB",
  ContainerName: "Corso_Restore_2022-09-23",
}
```

**Maybe expected:** For UserA and UserB, restore Inbox to folder1 in their separate hierarchies
```go
dest := selectors.RestoreDestination{
  ContainerName: "Corso_Restore_2022-09-23",
}
```

Given the limited nature of the destination requirement, a POD struct seems like the best option for the foreseeable future. If we need more complex functionality down the line we can expand from this

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #897 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-26 20:57:04 +00:00
Keepers
38addfaae8
track bytes written, read in kopia (#944)
## Description

Track the count of bytes read and written in
kopia.  For backups, this means the count of
bytes fed into kopia (hashed bytes), and the
amount written after compression and dedupe
(total file bytes).  For restore, this is the count of bytes in all files read.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #894

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-26 19:18:42 +00:00
ashmrtn
ff2db0c553
Export path package to other codebases (#912)
## Description

Moves the `path` package to the `pkg` package so other code outside of Corso can use it if they need it

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* closes #908 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-26 18:34:19 +00:00
ashmrtn
8c674fa17b
Create interface for Repository (#938)
## Description

Aid in testing. Break the interface in two for the moment, one of which is embedded in the other. The smaller interface focuses just on getting information about backups from the ModelStore.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-22 22:09:16 +00:00
Danny
3f3c8217a3
GC: Component: Version 40.0 (#917)
## Description
Upgrades Graph API to the newest version. Breaking change from the previous version, all functionality needs to be inspected to ensure operation.  
<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #828<issue>

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2022-09-22 18:59:51 +00:00
Keepers
76f2c5a632
improve language on selector reduce error (#794)
## Description

shorten and consolidate the error message when
the provided selectors match zero entries in the
backup entity list.

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #564

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-22 15:44:08 +00:00
Keepers
d633be9eef
nightly load test footprint and workflow (#928)
## Description

Adds the footprint file for running nightly load
tests out of the repository package.  The tests
don't do anything at this stage, but that's okay.
Also introduces a workflow to kick off the tests
every day at 2am UTC.

## Type of change

- [X] 🤖 Test
- [X] 💻 CI/Deployment

## Issue(s)

* #902

## Test Plan

- [X] 💚 E2E
2022-09-22 15:16:25 +00:00
ashmrtn
f2e767b3c7
Expand selectors package Reduce tests (#925)
## Description

Add test data and more complex tests for the selectors package. This helps codify the interface behavior of selectors which can help avoid surprises in the future

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

* #913 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-21 23:15:28 +00:00
Keepers
714e7e3a9b
quick test expansion for includesCategory (#914)
## Type of change

- [x] 🤖 Test

## Issue(s)

* #810

## Test Plan

- [x]  Unit test
2022-09-20 19:34:40 +00:00
Keepers
41bb3ee6f9
prevent cross-contamination on filter reduce (#898)
## Description

Over-restrictive scope correlation caused the
reduce processor to only include filters which
matched the data type of the scope.  Ironically,
this allowed a superset of information to match,
by evading the _all-match_ expectations of
filter scopes.

Also replaces the data category consts inside /details
with the path category types, since those are acting as
better canonical owners of data type identification
throughout the app.

## Type of change

- [x] 🐛 Bugfix

## Issue(s)

* #890

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
2022-09-20 00:34:51 +00:00
Keepers
a0508cc442
use account id, mock event bus (#870)
## Description

1/ introduce an ID() method int the account
package which provides the tenantID of the
configured account provider.

2/ introduce a event bus mock for testing.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #741

## Test Plan

- [x]  Unit test
2022-09-19 19:40:43 +00:00
Keepers
d983a77a57
normalize s3 bucket names (#847)
## Description

Allow users to provide bucket names prefixed
with the s3 scheme: s3://my-bucket

## Type of change

- [x] 🐹 Trivial/Minor

## Issue(s)

* #560

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-19 19:04:32 +00:00
Vaibhav Kamra
a9b0e2e7ee
OneDrive CLI commands (#739)
## Description

Wires up the OneDrive Backup(create,list,details,delete) and Restore commands

Unit tests added but integration tests will be added after the underlying operation
PRs are merged.

## Type of change

Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor

## Issue(s)
#658 
#668 

## Test Plan

<!-- How will this be tested prior to merging.-->

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-16 20:32:36 +00:00
ashmrtn
9e66f197c0
Create and populate folder backup details entries during backup (#872)
## Description

Other components may need to rebuild the directory hierarchy of items. As the paths Corso deals with can be hard to properly parse at times, store that information in the Corso backup details. The hierarchy can be rebuilt by following the `ParentRef` fields of items. The item at the root of the hierarchy has an empty `ParentRef` field.

Also hide these folders from end-users. They are not displayed during backup list nor are they eligible as a target for restore

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #862
* closes #861
* closes #818 

merge after:
* #869 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-16 01:42:01 +00:00
Keepers
1f4a490c33
allow users to opt-out of metric gathering (#871)
## Description

Adds a flag and options control for disabling metrics.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #741

## Test Plan

- [x] 💪 Manual
2022-09-16 00:56:56 +00:00
ashmrtn
ad2d046d19
Create folder struct for backup details (#869)
## Description

<!-- Insert PR description-->
Folder entries allow recording the directory hierarchy of a backup in backup details for other consumers.

Although folders are present in the backup details and they have the logic to be displayed, they are not expected to be shown to users (later PRs will ensure this).

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #860 

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-16 00:17:28 +00:00
Keepers
a6ba26cd92
generate events (#863)
## Description

Adds events for repo init, backup start and end,
and restore start and end.  Not all expected
values are available.  TODOs have been marked
for values that need future implementation.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* #741

## Test Plan

- [x]  Unit test
2022-09-15 18:03:20 +00:00
Keepers
ed52a07e2f
utilize shorthash in selector sels (#854)
## Description

The details entry shorthash can be treated as
equal to the leaf item ID.  This adds support to
the selector reduce step to compare the leaf val
to either the path item ID or the shortHash.

## Type of change

- [x] 🌻 Feature

## Issue(s)

* closes #572

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-15 16:53:03 +00:00
Keepers
e92cb1b62b
extend logging support for testing (#850)
## Description

Updating logging to help display/control within testing envs.
Restore error tracking within gc iterators.

## Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Test

## Issue(s)

* #824

## Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2022-09-15 13:18:05 +00:00
Keepers
0c8a03a826
add the metrics package (#839)
## Type of change

- [x] 🌻 Feature

## Issue(s)

* #741

## Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-09-15 00:28:02 +00:00
ashmrtn
4cf5fcab5a
Use short ref during backup (#851)
## Description

Short refs allow the user to type less to select an item to restore. This PR
* Adds a short ref field to the BackupDetails entries
* Populates the short ref field during backup

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #572

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-15 00:09:28 +00:00
ashmrtn
573f55686f
Make ExchangeDataCollection use path.Path internally (#820)
* Have exchange data collection store path.Path

Still complies with the old FullPath() string interface until we update
that.

* Pass path.Path to NewCollection for exchange

Basically fixes up errors introduced by previous commit.

* Fixup exchange recovery path indices

All exchange paths now use the path struct, meaning the service,
category, and user elements are in the standard positions.

* use path package in selector reduction (#822)

Currently, during a reduction process, scopes
compare their values to the raw split on repoRef.
This causes some brittle indexing to retrieve
values from the rr, carrying assumptions that are
difficult to track across changes.  This PR trades
the string split for the paths package to better
integrate identification of the path values.

Adds some mocks and amends some error
behaviors in order to fit paths into the current
testing schema.

Co-authored-by: Keepers <ryanfkeepers@gmail.com>
2022-09-13 13:58:20 -07:00
Vaibhav Kamra
16ad25f3a1
Implement wrapper around M365 helper methods (#825)
## Description

Exports functionality useful for external services/validators

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2022-09-13 10:32:02 +00:00