* Make data.Collection.FullPath return path.Path
* Fixup graph connector code for path struct
* Add Elements call to Path interface
Still should not be used in place of the named functions to get specific
path elements (e.x. ResourceOwner())
* Fixup kopia wrapper path handling
Mostly removing shim code and minor fixup for building the directory
tree.
* All the test fixes
* Helper function to append elements to a path
Kopia wrapper will need to create the complete path of an item by
joining the collection path and the item name. This allows it to do so
without having to drop to a path Builder (not service/category safe) and
go back to a resource path. Right now it does not handle escaping.
* Use path struct while streaming entries
Use new path struct while streaming entries to kopia. Preparation for
FullPath returning a path struct.
* Update tests to use valid path structures
## Description
Small refactor of the data collection restore helper so that we can plug-in the OneDrive helpers in a
follow-up PR
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
* Constants for OneDrive stuff
* Tests and constructor for OneDrive paths
* Populate onedrive path struct in data collection (#835)
* Helper function to make path structs for onedrive
* Use path struct in onedrive data collection
Does not change the external API at all, just the internals of how
FullPath functions and what is stored for the path.
* Wire up making data collections with path struct
Requires addition of tenant as input to Collections().
* Fixup onedrive Collections tests
* Wire up call to onedrive.NewCollections()
Just requires adding the tenant ID to the call.
* Use path struct in kopia DataCollection
Does not change the external API of the DataCollection any, just updates
internals in preparation for switching support of
data.Collection.FullPath.
* Expand Path interface slightly
kopia.Wrapper needs some extra functionality from paths, mostly along
the lines of directly manipulating the elements in the path. This gives
access to those functions.
* Use path struct in kopia.Wrapper for restore
Pass path structs to the newly created collections during restore.
* Add tests for new path functionality
* Have exchange data collection store path.Path
Still complies with the old FullPath() string interface until we update
that.
* Pass path.Path to NewCollection for exchange
Basically fixes up errors introduced by previous commit.
* Fixup exchange recovery path indices
All exchange paths now use the path struct, meaning the service,
category, and user elements are in the standard positions.
* use path package in selector reduction (#822)
Currently, during a reduction process, scopes
compare their values to the raw split on repoRef.
This causes some brittle indexing to retrieve
values from the rr, carrying assumptions that are
difficult to track across changes. This PR trades
the string split for the paths package to better
integrate identification of the path values.
Adds some mocks and amends some error
behaviors in order to fit paths into the current
testing schema.
Co-authored-by: Keepers <ryanfkeepers@gmail.com>
expands the `purge` command to accept args for
purging mail folders, calendars, or both. This
allows the test cleanup to ensure we aren't over-
populating either mail folders or calendars, thus
blocking CI actions
Allows constructing Contacts and Events category type paths for Exchange.
They all need user and tenant filled in and only differ in the category given.
This makes it easier to call them in more generic situations, so long as the
category is known. It also consolidates code for constructing the paths of those
types.
## Description
Adds foundational cli integration tests for backup/restore
of events and calendars
## Type of change
- [x] 🤖 Test
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
Single Legacy Policies added to allow the values for sent / receive times to correspond to the original content for `exchange.Mail` objects
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #645
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
Values can be inspected using e2e restore pipelines.
## Description
Pagination fix for nill pointer
## Type of change
- [x] 🐛 Bugfix## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#801
## Test Plan
- [x] ⚡ Unit test
## Description
Fixes mapping alignment during iterator tests
## Type of change
- [x] 🐛 Bugfix
- [x] 💻 CI/Deployment
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#797
## Test Plan
- [x] ⚡ Unit test
## Description
Enables a logging middleware that will log the Graph http calls if the `LOG_GRAPH_REQUESTS` environment variable
is set
## Type of change
Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
```$ LOG_GRAPH_REQUESTS=true CORSO_CI_TESTS=true go test -v .
...
=== RUN TestGraphConnectorIntegrationSuite
2022-09-08T13:39:25.556-0700 INFO graph/service_helper.go:61 REQUEST: GET /v1.0/users?%24select=id%2CuserPrincipalName HTTP/1.1
Host: graph.microsoft.com
Accept: application/json
Authorization: Bearer <REDACTED>
integration_runners.go:58: TestGraphConnectorIntegrationSuite run at 2022-09-08T20:39:25.967824Z
=== RUN TestGraphConnectorIntegrationSuite/TestAccessOfInboxAllUsers
2022-09-08T13:39:26.398-0700 INFO graph/service_helper.go:61 REQUEST: GET /v1.0/users?%24select=id%2CuserPrincipalName HTTP/1.1
Host: graph.microsoft.com
Accept: application/json
Authorization: Bearer <REDACTED>
```
## Description
Wires up the OneDrive collection logic to `operation.Backup`
Includes an integration test that runs against the test domain
Two bug fixes:
- Skip the "root" item that is returned by the delta query
- Fix incorrect usage of the `filepath.SplitList` function which does
not split a path into components. Instead use `strings.Split`. This
is ok because the paths returned here are not OS specific.
Regardless - this logic will be refactored when we use the `path` pkg.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
#548
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.
All other changes in the repository update the
package imports to the new module path.
The path package changed the standard format
of fullPath and repoRef design. This should have
failed tests before being pushed to main, but was
able to slip in falsely while github actions were
configured to pass all tests until failed.
## Description
Creates and uses a `ContainerResolver` interface to fetch container paths for items of different categories (when other resolvers are implemented). If the resolver is not available or fails to resolve a folder, defaults to the old implementation of using the folder's ID as its path.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #456
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Will be useful for parts of graph connector code that are abstracted to
take a category and return collections. Does not work for exchange
events or contacts yet as those paths haven't been defined.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
part of:
* #671
* #456
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Will help make later PRs easier as the category will already be known.
## Description
<!-- Insert PR description-->
## Type of change
Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
on path to:
* #456
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Instead of initializing static AWS credentials, we rely on the
credential provider chain in [Kopia](https://github.com/kopia/kopia/pull/2213) to discover and initialize credentials.
This currently supports the following in this order:
- Static credentials
- Environment variables (what Corso used to implement)
- IAM
Going forward, this will also allow us to support shared credentials (cred file) once that is
added to the credential provider chain.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
- #384
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual (see test output below)
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
This builds on the `MergeStatus` proposal and `WaitGroup` discussion proposed in #494
Required for OneDrive where we are operating on multiple collections
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
#494
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Turn into single constructor with bool instead of separate item and
folder constructors.
## Type of change
Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
#671
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add events and contacts. Still cannot create paths of these types
though.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
#671
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Group like parameters to graph functions so SonarCloud is happier
## Type of change
Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
on the path to:
* #456
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Will be used later and just generally useful if we ever add tracing or
such to these functions.
## Type of change
Please check the type of change your PR introduces:
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
#456
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Directory cache capable of converting a folder ID to a complete path to the folder
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
#456
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
When a mock event is created the event is sent to personnel. Requires that personnel have valid email addresses.
Addresses created and updated in the mockconnector package.
## Type of change
Please check the type of change your PR introduces:
- [x] 🐛 Bugfix
## Test Plan
- [x] 💪 Manual
No emails populate when `TestRestoreEvents` completes
## Description
Exchange events were amended to use calendars
as a folder structure. This updates the selector
to treat events as having folders similar to mail and
contacts.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
logic and orchestration to take byte representation of M365 event, create a Corso calendar, and place the event and calendar into the M365 user's account.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
closes#599closes#710
Requires PR #708 to ensure test folder is where events are created.
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] ⚡ Unit test
## Description
Since a backup can encapsulate multiple data
types (ex: mail, contacts, and events), it doesn't
make sense to print one table with all disjoint
values. This change splits up the print output
so that each data type in the details gets its
own table.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
* logic to split escaped path strings into elements
* wire up validity checks/struct construction
* tests for both invalid and valid path strings
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
part of #671
merge after:
* #648
* #689
* #690
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
This adds the logic that materializes collections for a specified user
The collection paths are currently derived from OneDrive metadata but a future
PR will introduce `paths` pkg support to create these.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
#388
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
* Logic to validate service/category pairs for paths
* convert string to service or category const
* check that a given service/category pair is a valid resource type
* Add basic tests for validateServiceAndCategory
* Re-add logic for getting Exchange mail paths
Paths can either be for an item (email) or a folder. Returned struct
provides safe access to underlying information in the path once it is
created.
* Refine tests, add struct comment
Add test to make sure Item() and Folder() work properly for a resource
that has no parent folder, just the prefix stuff.
* Rework resource-specific paths based on comments
* use a single type to represent resource specific paths
* use service/category enums to denote the resource the path represents
* update tests to check for service/category of the path
* rename exchange_path*.go -> resource_path*.go
