## Description
Use multiple restores to populate the account with known data and then run a backup over the restores to pull the data.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Test restoring a single contacts folder and multiple contacts folders. This tests that items from different folders can be restored, but not that items from multiple folders can be backed up (because the restore colocates into a single folder)
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
merge after
* #986
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Stop fetching all folders for every new folder seen, erasing previous results. Regression test to come in a later patch as it requires some extra infrastructure
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#982
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Implementation of simplified Restore based on the following [spec](https://www.notion.so/alcion/Event-restore-semantics-061aee5288244629b1c53337e4dea306#6e974540c8804c4fa832218675534e1c)
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #954<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
## Description
* treat empty values and nil pointers as equal when comparing Graph models
* make sure all items are always iterated through so the test doesn't hang
* explicitly check number of items retrieved
* remove some unneeded casts and assignments
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds some go runtime diagnostics tracking to
load testing, including some trace regioning.
Unfortunately, I couldn't find any third party trace library that didn't depend on a sidecar server
to sample against the application on. Therefore,
just starting with something basic.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Test bad services or empty collections don't cause an error
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add framework and tests for some exchange mail restore and backup situations. Framework can be used to test other situations in the future, this is just the starting point.
Some logic in the test can be further generalized/factored out once we know more about how paths will be transformed during restore
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
The feature ensures that this Flat style directory only creates a single folder and restores all the contacts to the same folder during the restore process.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #880<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds the ability to use mockconnector for onedrive and exchange restores, reducing potential code duplication overall.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
* No new functionality exposed to CLI users
* generate restore folder names in CLI and pass down the stack
* update tests for new parameters
* centralize generation of restore container names
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #897
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Exchange app e2e tests need to scope their
backup dataset to only the primary folder, else
we risk exponential duplication of data when
the next backup picks up all the corso-copied
folders.
## Type of change
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
* #501
### Awaiting completion
- #810
- #811
## Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
Mockdata exchanged for a newer version Kiota serialized version.
## Description
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #956<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Addresses a follow-up from the folder lookup implementation where we were not iterating through
all children (no pagination was implemented) when searching for a folder.
Instead of implementing pagination, this uses OneDrive path based addressing to do a direct lookup
for the folder.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #875
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds file size and timestamps to enable selectors using this metadata
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #594
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Fixes a couple of regressions that got introduced on the restore path via code review changes and
kopia wrapper refactor.
Found while writing the OneDrive restore test. Would have been prevented if we had said test.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Moves the `path` package to the `pkg` package so other code outside of Corso can use it if they need it
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#908
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Feature branch enabling filtering based upon `selectors.ExchangeScope`
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #810<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Moves constants to a file and adds more source definitions and backgrounds to constants.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #823<issue>
## Test Plan
Const variables to be used in future PRs. No tests are available.
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Upgrades Graph API to the newest version. Breaking change from the previous version, all functionality needs to be inspected to ensure operation.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #828<issue>
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Finish enabling wsl linter on Corso repo by removing the exception for onedrive package
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* closes#481
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds function call to the path package that returns the folder elements present in a path after the path prefix. This is safer than using `strings.Split()` because some paths may have `/` characters in elements.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#909
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Implements `data.Collection` restore for OneDrive and wires it up to the restore operation
Leverages previously added `onedrive` helpers that have integration tests.
An op/CLI level integration test will be added in a follow-up PR.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #668
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
```
$ ./corso restore onedrive --backup 7de4d68f-67c2-4e81-95a2-81cf0d1bd11d
Restored OneDrive in S3 for user
```
## Description
PR enables Filtering for `exchange.Contact` items wherein if an `exchange.ContactFolder` is requested specifically, only the requested folders are backed up.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #811<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Small refactor. Moves Path functions from iterator to `service_functions.go`
Introduces getCollectionPath which combines the two previous functions and reduces code complexity
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
This implements helper methods that support creating new items and uploading data from a `io.Reader` to OneDrive.
It includes an integration test that uses mock data.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #668
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds the ability to retrieve M365 IDs of contacts not in the default folder.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#804
* closes #767<issue>
* closes#669
## Test Plan
- [x] ⚡ Unit test
## Description
Updating logging to help display/control within testing envs.
Restore error tracking within gc iterators.
## Type of change
- [x] 🐛 Bugfix
- [x] 🤖 Test
## Issue(s)
* #824
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Places `exchange.Restore` logic into single file
<!-- Insert PR description-->
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Stops the generation of Inbox messages when events are created
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #855<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
Verify that Inbox doesn't grow with undeliverable messages.
* Make data.Collection.FullPath return path.Path
* Fixup graph connector code for path struct
* Add Elements call to Path interface
Still should not be used in place of the named functions to get specific
path elements (e.x. ResourceOwner())
* Fixup kopia wrapper path handling
Mostly removing shim code and minor fixup for building the directory
tree.
* All the test fixes
## Description
Small refactor of the data collection restore helper so that we can plug-in the OneDrive helpers in a
follow-up PR
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
* Constants for OneDrive stuff
* Tests and constructor for OneDrive paths
* Populate onedrive path struct in data collection (#835)
* Helper function to make path structs for onedrive
* Use path struct in onedrive data collection
Does not change the external API at all, just the internals of how
FullPath functions and what is stored for the path.
* Wire up making data collections with path struct
Requires addition of tenant as input to Collections().
* Fixup onedrive Collections tests
* Wire up call to onedrive.NewCollections()
Just requires adding the tenant ID to the call.
* Have exchange data collection store path.Path
Still complies with the old FullPath() string interface until we update
that.
* Pass path.Path to NewCollection for exchange
Basically fixes up errors introduced by previous commit.
* Fixup exchange recovery path indices
All exchange paths now use the path struct, meaning the service,
category, and user elements are in the standard positions.
* use path package in selector reduction (#822)
Currently, during a reduction process, scopes
compare their values to the raw split on repoRef.
This causes some brittle indexing to retrieve
values from the rr, carrying assumptions that are
difficult to track across changes. This PR trades
the string split for the paths package to better
integrate identification of the path values.
Adds some mocks and amends some error
behaviors in order to fit paths into the current
testing schema.
Co-authored-by: Keepers <ryanfkeepers@gmail.com>
expands the `purge` command to accept args for
purging mail folders, calendars, or both. This
allows the test cleanup to ensure we aren't over-
populating either mail folders or calendars, thus
blocking CI actions
## Description
Adds foundational cli integration tests for backup/restore
of events and calendars
## Type of change
- [x] 🤖 Test
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
Single Legacy Policies added to allow the values for sent / receive times to correspond to the original content for `exchange.Mail` objects
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #645
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
Values can be inspected using e2e restore pipelines.
## Description
Pagination fix for nill pointer
## Type of change
- [x] 🐛 Bugfix## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#801
## Test Plan
- [x] ⚡ Unit test
## Description
Fixes mapping alignment during iterator tests
## Type of change
- [x] 🐛 Bugfix
- [x] 💻 CI/Deployment
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#797
## Test Plan
- [x] ⚡ Unit test
## Description
Enables a logging middleware that will log the Graph http calls if the `LOG_GRAPH_REQUESTS` environment variable
is set
## Type of change
Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
```$ LOG_GRAPH_REQUESTS=true CORSO_CI_TESTS=true go test -v .
...
=== RUN TestGraphConnectorIntegrationSuite
2022-09-08T13:39:25.556-0700 INFO graph/service_helper.go:61 REQUEST: GET /v1.0/users?%24select=id%2CuserPrincipalName HTTP/1.1
Host: graph.microsoft.com
Accept: application/json
Authorization: Bearer <REDACTED>
integration_runners.go:58: TestGraphConnectorIntegrationSuite run at 2022-09-08T20:39:25.967824Z
=== RUN TestGraphConnectorIntegrationSuite/TestAccessOfInboxAllUsers
2022-09-08T13:39:26.398-0700 INFO graph/service_helper.go:61 REQUEST: GET /v1.0/users?%24select=id%2CuserPrincipalName HTTP/1.1
Host: graph.microsoft.com
Accept: application/json
Authorization: Bearer <REDACTED>
```
## Description
Wires up the OneDrive collection logic to `operation.Backup`
Includes an integration test that runs against the test domain
Two bug fixes:
- Skip the "root" item that is returned by the delta query
- Fix incorrect usage of the `filepath.SplitList` function which does
not split a path into components. Instead use `strings.Split`. This
is ok because the paths returned here are not OS specific.
Regardless - this logic will be refactored when we use the `path` pkg.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
#548
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
