## Description
Store and retrieve owner info for onedrive files.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Introduces config options (`--disable-tls` and `--disable-tls-verification`) to turn off TLS and TLS verification
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1415
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
As the exchange container resolvers are pretty close implementation-wise, consolidate the generic code. Also move some of the tests around to better fit what file has the code being tested
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1362
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add tags to each kopia snapshot that include all service/category pairs in the snapshot and all resource owners in the snapshot. This allows future snapshots to lookup existing snapshots by those tags so they can be fed into the snapshot function. Feeding previous snapshots into the snapshot function enables kopia to detect previously uploaded files and skip uploading the data again
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1404
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Grouping of event and contact folder caches to the same file. MailFolderCache left in a different folder until it can be decided where the tests before
<!-- Insert PR description-->
## Type of change
- [x] 🤖 Test
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
The panic in development mode is interfering with test scrutiny and other general behavior. Errors will be reported naturally without the need for this additional panic.
## Type of change
- [x] 🤖 Test
- [x] 🐹 Trivial/Minor
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Adds feature to be able to create mock events with attachments
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1005 <issue>
## Test Plan
Testing:
```
internal/connector/exchange$ go test -v . --run TestExchangeServiceSuite/TestRestoreExchangeObject/Test_Event_with_Attachment
```
- [x] ⚡ Unit test
## Description
Add aws xray observability tooling, plus ways to quickly and
easily run the tools/load tests. Next steps are 1/ running the
daemon container alongside the load tests within github, and
2/ creating client credentials in aws's xray service so that
gathered trace info is propagated from the daemon proxy
to the xray service.
## Type of change
- [x] 🤖 Test
- [x] 💻 CI/Deployment
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Use the corso logger in analytics, to allow control
over log outputs from that package.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1324
## Test Plan
- [x] 💪 Manual
## Description
Adds production of 10 exchange items of each type, only for the load_test user, every time the load tests succeed. Also migrates that user ID out to a secret value stored in github.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Builds a scope.Category() -> path.CategoryType trans- former func into scopes to standardize the relationship between the two properties.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1133
## Test Plan
- [x] ⚡ Unit test
## Description
Command line utility that allows for the printing of mockData based on the m365ID. Supports:
- `exchange.Mail`
- `exchange.Contacts`
- `exchange.Events`
<!-- Insert PR description-->
## Type of change
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
Related to:
* #1255<issue>
* #999
* #705
## Description
This adds the following functionality to the CLI progress output:
- Add a message e.g. to describe a completed stage
- Add a message describing a stage in progress (using a spinner) and trigger completion
- Add a progress tracker (as a counter) for a specified number of items (when the # items is known up front
- Improves `ItemProgress` by aligning the columns a bit better and also removing the progress bar and replacing
with a byte counter
Finally - the above are used in the `backup` and `backup create onedrive` flows. Follow up PRs will wire these up for
exchange and the restore flows also.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1278
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Test flakes because CI cleanup job goes through and deletes some folders the folder cache ends up looking for. Solve this by only selecting specific mail folders instead of all of them.
Will need to rely on other tests to check if fetching all mail folders works. As only some folders are checked for though, overall impact of removing this test should be minor as the more targeted test is still present
Mistakenly re-added by #1134
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1222
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Basic smoke tests to ensure multi-user restore and backup work properly. Will need updated to add OneDrive tests later on once the OneDrive test pattern is merged
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
merge after
* #1186
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a flag that allows users to toggle off the progress
bar spinners. Primarily useful when running in a lower
log level such as debug.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
## Description
verifyBackupInputs() is a newly created function that ensures that a selector given to GraphConnector is valid to continue.
Test src file: `graph_connector_test.go` expanded.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1258<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
<!-- Insert PR description-->
## Type of change
- [x] 🗺️ Documentation
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*Related to #1225<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
## Description
Switches the test accounts used in CI since the previously used account was hitting slowness due to
items in Deleted Folders.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1191
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds a tag describing the service within the backup model and the backup details model. Adds a filter builder pattern to the store wrapper for filtering results according to certain tags. Finally, adds a filter builder to specify the service type when listing backups.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1226
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
`path` is not cross platform as it hardcodes / as the separator. Also add a lint rule for this.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/1236
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Code removal from GraphConnector
`AddItemsToCollection` removed from graph_connector.go
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
`graph.ContainerResolver` has the capacity to keep the directory structure of the m365 objects that are helpful within a user's account. Leveraging this abstraction allows for a better flow of data from M365 into storage.
## Type of change
- [x] ⚡ : Optimization
## Issue(s)
* closes #1125<issue>
* closes#1122
## Test Plan
- [x] ⚡ Unit test
## Description
Adds a progress spinner tracking the collection-level work on an incremental basis. Useful for tracking collection progress when individual files may be too quickly resolved to visibly show progress bars.
In the future, we could combine the two so that the current item in progress always displays below the collection tracker.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
There's enough other tests in the codebase that check the functionality of getting multiple folders that it should be fine without this one.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1222
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Create a lint rule that warns if `context.TODO` or `context.Background` are called in tests. This helps standardize the use of `tester.NewContext` and ensures log messages are properly flushed during tests.
Output from the linter is similar to (sadly can't get rid of the `\\` in there)
```
pkg/services/m365/m365_test.go:49:22: use of `context.Background` forbidden because "tests should use tester\\.NewContext" (forbidigo)
ids, err := UserIDs(context.Background(), acct)
```
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1198
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
purge.go to use the ContainerResolvers to collect containers from the Graph Connector.
NOTE: CollectFolders in service_query.go unchanged. `CollectFolderrs()` is stubbed for another PR. This will reduce the amount of line changes to be used in the future.
<!-- Insert PR description-->
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1189<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Corso should be able to have `Backup` called on a user that does not have OneDrive or does not have OneDrive files without producing an error. This feature branch creates OneDriveIntegration testing to be able to fetch OneDrive data for a particular user.
Logic changed within `drive.go`
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
**closes** #966
## Test Plan
- [x] ⚡ Unit test
## Description
Adds the capacity to programmatically generate events in the factory script. Status is currently functional in code completion, but experiencing unexpected results.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Just code cleanup
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1156
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Wire up code to fetch secondary user from config/env and also provide it to tests.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1185
merge after to ensure automatic cleanup:
* #1193
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
* switch up email restore/backup test slightly to cover multiple folders
* remove multi-folder mail backup, now overlaps with the above
* remove stale TODO
* remove unused struct field
* factor out common struct
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Removal of deprecated functions from code base.
- GetRestoreContainer no longer used for restore workflow
- GetContainerID no longer used to find the M365ID of container objects.
## Description
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
In the event that a user only has a primary contact
folder, and no subfolders, the contact folder legacy
iter needs to fall back to checking for the contacts
default folder in an isolated query, because that
folder isn't provided as part of the contacts folders
get request.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1113
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Helper code for getting output destination has been merged so it's possible to reenable these tests now.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1169
merge after:
* #1184
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
M365 overrides recipient names if the email is known so don't compare that while checking.
Similar updates will likely be needed for other fields like [B]CC in the future.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1183
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.
⚠️ Your local environment will need to change in the following ways: ⚠️
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #558
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds scripted production of mock emails for building out
large/rolling datasets for load testing.
## Type of change
- [x] 🌻 Feature
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
## Description
Feature to add the folder hierarchy for folders when restored.
This required an overhaul of the `graph.ContainerResolver` interfaces:
- MailFolderCache
- ContactFolderCache
- ~EventFolderCache (placed in a separate PR)~ https://github.com/alcionai/corso/pull/1101
Restore Pipeline changed to separate the caching / container creation process from the rest of the restore pipeline.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
* closes#1046
* #1004
* closes#1091
* closes#1098
* closes#1097
* closes#1096
* closes#1095
* closes#991
* closes#895
* closes#798
## Test Plan
- [x] ⚡ Unit test
## Description
This sets up the metrics configs using `ldflags`. Keeping it in draft as I wanted to wait till https://github.com/alcionai/corso/pull/1052 is merged as that will affect how we use it in the CI. I have currently set the base branch to `release-ci` on GH, the branch for https://github.com/alcionai/corso/pull/1052 instead of `main` as the diff would make more sense that way.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1067
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Mostly path manipulations here to allow creating selectors for a backup and paths for the output of a backup given some information about the input. Capable of working for OneDrive and Exchange.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Extends progress bar display with multi-line support.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Use an upload session and chunked writes for large attachments.
This commit moves the logic used for OneDrive file upload to the common `uploadsession` pkg and
leverages that for the upload.
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1115
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
