## Description
Adds Created and Modified time to all objects. Need to go after
https://github.com/alcionai/corso/pull/1446 as it is built on top of
that. Keeping in draft until then.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
Adds the sharepoint service to /pkg/path. Currently uses the
"Files" category for its category type, which is just a placeholder
for kicking off development.
Additionally, uncomments selector tests that were dependent
upon the path service declaration.
## Description
Adds the boilerplate for sharepoint selectors
to the selectors package. Many categories are
given filler names (folders, items) and are
expected to change through development as we
pin down the design for those structures.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1462
## Test Plan
- [x] ⚡ Unit test
## Description
Allows users to specify whether they want to
select folders by prefix or search behavior.
Search/contains behavior is the default case,
with prefix being an optional deviation if the
folder input is prepended with a '/' character.
Also, propagates the PrefixMatch setting to
all integration tests that rely on selecting only
the default folder in exchange.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1224
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
For folder-level scopes (ie, scopes that compare
folder-hierarchy path segments), this change
replaces the standard "equals" and "prefix"
string comparators with the new PathContains
and PathPrefix comparators.
Next change is to interpret user inputs in the
cli to determine whether the comparator should
use contains or prefix behavior.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1224
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Creates Sharepoint Info within the details package. SharepointInfo constructor and test package included
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Test Plan
- [x] ⚡ Unit test
## Description
Add EndTime to Exchange events. Also fixes a testcase around start time.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Three primary changes:
1. move the list of load test users into a secret in the github test domain.
2. purge all data that was generated for users in the load test
3. fix all `m356` typos.
## Type of change
- [x] 🤖 Test
- [x] 💻 CI/Deployment
## Issue(s)
* #1266
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Store and retrieve owner info for onedrive files.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1366
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds two specialized filters: path prefix and path
contains. These filters differ from prior additions
in two ways: 1/ they accept a slice of inputs and
perform an implicit any-match when compared,
2/ they have particular rules about matching
completely within delimited elements.
In the next PR, these filters will replace the
standard prefix and contains comparisons that
are currently used in folder selection.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1224
## Test Plan
- [x] 💚 E2E
## Description
Introduces config options (`--disable-tls` and `--disable-tls-verification`) to turn off TLS and TLS verification
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1415
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Add aws xray observability tooling, plus ways to quickly and
easily run the tools/load tests. Next steps are 1/ running the
daemon container alongside the load tests within github, and
2/ creating client credentials in aws's xray service so that
gathered trace info is propagated from the daemon proxy
to the xray service.
## Type of change
- [x] 🤖 Test
- [x] 💻 CI/Deployment
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Use the corso logger in analytics, to allow control
over log outputs from that package.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1324
## Test Plan
- [x] 💪 Manual
## Description
Adds production of 10 exchange items of each type, only for the load_test user, every time the load tests succeed. Also migrates that user ID out to a secret value stored in github.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Builds a scope.Category() -> path.CategoryType trans- former func into scopes to standardize the relationship between the two properties.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1133
## Test Plan
- [x] ⚡ Unit test
## Description
Intercept the lower-level RepoAlreadyExists error, and field a repository package version of the same error for sdk consumers to identify when initializing corso repos.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1136
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Backup load tests are working successfully, we can
un-skip the restore portion of load tests now.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Show messages for specific user interactions which previously had none.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/1177
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds a flag that allows users to toggle off the progress
bar spinners. Primarily useful when running in a lower
log level such as debug.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
## Description
Remove old "restored..." confirmation line from the CLI. Renamed the "Reference" column header to "ID" in the tabular output, which represents the shortRef aka item id.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1269
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Adds a tag describing the service within the backup model and the backup details model. Adds a filter builder pattern to the store wrapper for filtering results according to certain tags. Finally, adds a filter builder to specify the service type when listing backups.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1226
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Call the OneDrive flag validation function to ensure flags that take time strings are formatted properly. Also add tests to ensure there's no future regressions for this
First commit has important code changes. Remainder of commits have test data/setup
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1231
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Notes
All users in the test set have been ensured to have a
complete set of populated data, relevant to the per-
category-per-user assertions in the test.
## Type of change
- [x] 🐛 Bugfix
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Some minor bugs. Further assurance that users will appear
in the test results as expected will require manual curation
or addition of data in each user's account.
## Type of change
- [x] 🐛 Bugfix
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Create a lint rule that warns if `context.TODO` or `context.Background` are called in tests. This helps standardize the use of `tester.NewContext` and ensures log messages are properly flushed during tests.
Output from the linter is similar to (sadly can't get rid of the `\\` in there)
```
pkg/services/m365/m365_test.go:49:22: use of `context.Background` forbidden because "tests should use tester\\.NewContext" (forbidigo)
ids, err := UserIDs(context.Background(), acct)
```
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1198
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Remove JSON [de]serialization from mock model store and just use assignments. Model Store tests check that all fields are properly filled out so using JSON here is not required.
Unblocks #1089 as mock model store no longer relies on JSON serialization (and tags associated with models)
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1085
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.
⚠️ Your local environment will need to change in the following ways: ⚠️
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #558
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Exposing user IDs in addition to user emails.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [X] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [X] ⚡ Unit test
- [ ] 💚 E2E
## Description
Use paths package to trim unescaped trailing '/' characters from input for Exchange mail and OneDrive folder names. Add tests for Exchange showing that the trimming also works properly if the folder name ends with '/'.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1147
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Watch the minimum-complete load test over the weekend.
Backup only, all tenant users, all data types, no restrictions.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Added an additional set of test suites to load_test
to focus on the backup of a single user who will
have their account prepared with a significant amount
of data (many emails, many events, large files, etc).
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Adds a check that reviews the load test details.
Details entries to make sure that all users were
involved in the operation, and that for each user,
the test involved all item categories for that app.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
adds extensible options to folder-level scopes that allows the caller to specify whether they want a
prefix-comparison matcher or a contains-comparison matcher.
Also corrects the behavior of the prefix filter so that it accurately follows the "target is prefix of input" specification, rather than the reverse.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1133
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Repo `--prefix` values should be normalized with a trailing `/`
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* Fixes#1152
## Test Plan
<!-- How will this be tested prior to merging.-->
- [] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Some paths can generate identical unescaped strings even though they have different elements. Add a test ensuring they generate unique ShortRefs even under these conditions.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1092
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context. The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton. But in order for the logs to show up
within the test, they need to get flushed.
See my comments for focus on non-chaff changes.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Adds validation of cli inputs for time-based flags.
All time values that can be parsed by the time
handling in common count as supported, even
though official support includes a smaller set.
Also adds some clean-up to time.go and adds a
design document describing standard time value
formats, and the maintenance thereof, in corso.
## Type of change
- [x] 🌻 Feature
- [x] 🗺️ Documentation
- [x] 🤖 Test
## Issue(s)
* #943
* #924
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Load Testing is causing folder growth (fix is underway)
and is already taking > 4 hours only to back-up emails.
This reduces the dataset to only the primary folders for
each exchange type. We can expand from here when
data production is better under control.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
\~Only affects folders created by tests\~
Create and use helper function that sets the restore destination folders to have microsecond granularity. This is to avoid naming collisions with second granularity (of which I've observed at least once)
Switches all tests for all services/categories to use OneDrive DateTime format+microseconds
Manually verified that a manually created folder with the given timestamp format was removed by the CI purge script
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1036
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Now that GC is tracking the details entries during
a restore procedure, it can display the results in
the cli. Due to an absence of itemInfo, only the
item shortRef is displayed. But we can expand
on that from here.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #977
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
- Rename `LastModified` to `Modified` in OneDrive backup details
- Humanize file size units
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #627
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Ensure s3 bucket name inputs are normalized
in all uses. Currently, the normalized format is
applied when using storage, but not when passing
around --bucket flag values elsewhere.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #975
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Temporarily up the frequency of the load test kickoff for observation.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Adds the following selectors to OneDrive details/restore :
- `file-name`, `folder`, `file-created-after`, `file-created-before`, `file-modified-after`, `file-modified-before`
Also includes a change where we remove the `drive/<driveID>/root:` prefix from parent path entries in details. This
is to improve readability. We will add drive back as a separate item in details if needed later.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #627
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Specifying a folder on the CLI should cause the system to do a prefix match on that folder when doing a restore, backup details, or selector reduce operation.
Recently updated behavior, adjusting tests to match new expectations.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds some go runtime diagnostics tracking to
load testing, including some trace regioning.
Unfortunately, I couldn't find any third party trace library that didn't depend on a sidecar server
to sample against the application on. Therefore,
just starting with something basic.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Adds support for folder and item selection
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #627
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
