## Description
Adds the capacity to programmatically generate events in the factory script. Status is currently functional in code completion, but experiencing unexpected results.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Just code cleanup
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1156
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Remove JSON [de]serialization from mock model store and just use assignments. Model Store tests check that all fields are properly filled out so using JSON here is not required.
Unblocks #1089 as mock model store no longer relies on JSON serialization (and tags associated with models)
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1085
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Wire up code to fetch secondary user from config/env and also provide it to tests.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1185
merge after to ensure automatic cleanup:
* #1193
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
* switch up email restore/backup test slightly to cover multiple folders
* remove multi-folder mail backup, now overlaps with the above
* remove stale TODO
* remove unused struct field
* factor out common struct
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Removal of deprecated functions from code base.
- GetRestoreContainer no longer used for restore workflow
- GetContainerID no longer used to find the M365ID of container objects.
## Description
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
In the event that a user only has a primary contact
folder, and no subfolders, the contact folder legacy
iter needs to fall back to checking for the contacts
default folder in an isolated query, because that
folder isn't provided as part of the contacts folders
get request.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1113
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Helper code for getting output destination has been merged so it's possible to reenable these tests now.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1169
merge after:
* #1184
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
M365 overrides recipient names if the email is known so don't compare that while checking.
Similar updates will likely be needed for other fields like [B]CC in the future.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1183
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.
⚠️ Your local environment will need to change in the following ways: ⚠️
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #558
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Exposing user IDs in addition to user emails.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [X] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [X] ⚡ Unit test
- [ ] 💚 E2E
## Description
Use paths package to trim unescaped trailing '/' characters from input for Exchange mail and OneDrive folder names. Add tests for Exchange showing that the trimming also works properly if the folder name ends with '/'.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1147
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Watch the minimum-complete load test over the weekend.
Backup only, all tenant users, all data types, no restrictions.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Adds scripted production of mock emails for building out
large/rolling datasets for load testing.
## Type of change
- [x] 🌻 Feature
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
## Description
Added an additional set of test suites to load_test
to focus on the backup of a single user who will
have their account prepared with a significant amount
of data (many emails, many events, large files, etc).
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Introduces a `factory` script to help devs generate
fake m365 data in bulk for testing purposes. Only
boilerplate at this time. Any TODO's or not-yet-
implemented statuses are present as placeholders
alone, without official deadlines.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
Adds a processor that confirms whether user has
added a value for a flag in the cmd, or if it is the
default value. This map of valued flags is added
to the service opts structs to for validation.
Also migrates many service flags to utils as
consts so that these values can be maintained
as consistent across packages.
## Description
Feature to add the folder hierarchy for folders when restored.
This required an overhaul of the `graph.ContainerResolver` interfaces:
- MailFolderCache
- ContactFolderCache
- ~EventFolderCache (placed in a separate PR)~ https://github.com/alcionai/corso/pull/1101
Restore Pipeline changed to separate the caching / container creation process from the rest of the restore pipeline.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
* closes#1046
* #1004
* closes#1091
* closes#1098
* closes#1097
* closes#1096
* closes#1095
* closes#991
* closes#895
* closes#798
## Test Plan
- [x] ⚡ Unit test
* Make corso respect --version
* Add proper version information when building
* Use unified version string
$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)
With tag: v0.0.0-ceaf04c9
Without tag: unreleased-ceaf04c9
* Add a simple build command to build using Makefile
## Description
This sets up the metrics configs using `ldflags`. Keeping it in draft as I wanted to wait till https://github.com/alcionai/corso/pull/1052 is merged as that will affect how we use it in the CI. I have currently set the base branch to `release-ci` on GH, the branch for https://github.com/alcionai/corso/pull/1052 instead of `main` as the diff would make more sense that way.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1067
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Mostly path manipulations here to allow creating selectors for a backup and paths for the output of a backup given some information about the input. Capable of working for OneDrive and Exchange.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Extends progress bar display with multi-line support.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Adds a check that reviews the load test details.
Details entries to make sure that all users were
involved in the operation, and that for each user,
the test involved all item categories for that app.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
adds extensible options to folder-level scopes that allows the caller to specify whether they want a
prefix-comparison matcher or a contains-comparison matcher.
Also corrects the behavior of the prefix filter so that it accurately follows the "target is prefix of input" specification, rather than the reverse.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1133
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Allows listing individual backups, instead of always listing the entire set. Could be expanded to accept multiple backup IDs in the future.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1077
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Use an upload session and chunked writes for large attachments.
This commit moves the logic used for OneDrive file upload to the common `uploadsession` pkg and
leverages that for the upload.
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1115
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Repo `--prefix` values should be normalized with a trailing `/`
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* Fixes#1152
## Test Plan
<!-- How will this be tested prior to merging.-->
- [] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Hacks in the bytes written as the iteminfo size during onedrive item restores. Would use the drive itemable, but the size property isn't instantiated locally during creation, and cannot be mutated afterwards.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1113
## Test Plan
- [x] 💪 Manual
* add bool format checks for string flags
Adds a boolean-parseable validator for cli flags
that accept strings representing boolean values.
* fix ctx production in cli integration test
* Move interfaces to common location
Upcoming PRs are using these interfaces across packages. Move them to a
common location so multiple packages can use them without import cycles
etc.
* Allow adding newly created folders to the cache (#1131)
* New function to add folders to cache
Allow adding new folders to the cache. Automatically cache the paths for
the new folders. Also add the new function to the interface.
* Reuse the AddToCache function during population
* Wire up ability to back up a single subfolder of mail (#1132)
* Expand cache to return items in it
Required to allow matching an item's path to a selector as the selector
will not provide which paths it matches on easily.
* Function to get collections from cached folders
Returned collections match any matchers given for the folders
* Thread resolver through iterator functions
Allow the folder resolver to be used in all iterator functions. The
resolver will be tied to the current category and user.
* Choose between using resolver and making queries
Allow either using the resolver to get folders with matching names or
using queries to get them.
* Wire up resolver at entry point
Create a resolver instance for each user/category of data being
backedup.
* Preparation for changing how mail enumeration is done (#1157)
* Step towards redoing mail fetching
Pull out old way to get data into a new function and setup some helper
functions etc.
* Switch to pulling mail items folder by folder (#1158)
* Function to pull mail items given collections
Given a set of collections and IDs for those collections pull the mail
items for each collection.
* Create helper function to fetch mail
New helper function to fetch mail items. This goes through each folder
and gets the items for them individually.
* Wire up new way to fetch mail
Leaves fetch logic for other data types undisturbed.
* Tests for new mail fetching logic
Remove tests that were previously in iterators_test.go and move them to
graph_connector_test.go. These tests only had to do with mail logic.
Tests that handled all data types in iterators_test.go have been updated
to skip mail now.
* Add functions for OneDrive purge
function is setup to return and take purgable instead of just userID,
itemID for deleter.
* Adjust other functions to take purgable
Adjust interface for deleter so that it allows more information to the
delete function. The called function can cast the purgable it was passed
back to the type that was returned by the getter function.
* Move struct definition
Per reviewer comment move struct definition to OneDrive package instead
of leaving in purge command.
* Fix lint error
Likely slipped through while GitHub actions was having issues.
## Description
Wrong service referenced in restore output
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1099
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a new package- Observe- for owning user-
oriented displays like progress bars. This PR adds
an initial progress bar to onedrive backups as a
proof-of-concept. The API is more important than
the specific progress bar package at this time.
Future changes may opt for a different pkg.
Display format currently looks like:
```
59% [=============> ] (6.9/12 kB, 14 MB/s) | Item_Name.txt
```
Known Issues:
* the `progressbar` package does not support multiline output, and [the author is not planning to add support](https://github.com/schollz/progressbar/issues/6). This causes concurrent items to overwrite each other. We will either need to fork the library, or change to a different one.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Some paths can generate identical unescaped strings even though they have different elements. Add a test ensuring they generate unique ShortRefs even under these conditions.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1092
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context. The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton. But in order for the logs to show up
within the test, they need to get flushed.
See my comments for focus on non-chaff changes.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Contact Iterative sequence tests changed to reflect `exchange.Backup` functions used within `SetupExchangeCollectionVars()`. The previous bug was caused by misalignment from the old test configuration and the current implementation.
<!-- Insert PR description-->
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1107<issue>
## Test Plan
- [x] ⚡ Unit test
* Setup CI for release
* Initial commit of CHANGELOG.md
* Pin vale and markdownlint-cli to current versions
* Update branch name for CI
* Pull request workflow job
## Description
Useful helper functions for the purge script
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1090
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
CI test timing during times of high activity. Reduced test suite to cover Inbox and adjusted outputs accordingly.
<!-- Insert PR description-->
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1078<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
* Fill in developer/build.md page placeholder
* Fix some liniting issues
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds validation of cli inputs for time-based flags.
All time values that can be parsed by the time
handling in common count as supported, even
though official support includes a smaller set.
Also adds some clean-up to time.go and adds a
design document describing standard time value
formats, and the maintenance thereof, in corso.
## Type of change
- [x] 🌻 Feature
- [x] 🗺️ Documentation
- [x] 🤖 Test
## Issue(s)
* #943
* #924
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
purge restored folders for all users in the tenant following a load test to ensure we don't over-
produce data that causes future tests to time out.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #1048
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Currently only the ModelStore populates and checks versions.
Some high-level points:
* fail any sort of get (just metadata or full model) if there's a version mismatch
* update operations automatically set things to the current version
* versions are stored as tags so there's some int->string (and vice versa) munging
* versions stored as tags so they can be repopulated even if only the metadata (BaseModel) for a model is pulled. This is done mostly for consistency
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#284
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Removes the -all flag from the exchange cli. The
same functionality can be gained with the flag
--user *.
Additionally does some minor tidying of other
comments and outputs throughout the cli.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1032
## Test Plan
- [x] ⚡ Unit test
## Description
Load Testing is causing folder growth (fix is underway)
and is already taking > 4 hours only to back-up emails.
This reduces the dataset to only the primary folders for
each exchange type. We can expand from here when
data production is better under control.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Test to check that attachments are properly populated and fetched during restore and backup
Some notes:
* make slice element comparison function generic so it can compare locations and attachments
* very hacky comparison for attachment content using reflection as it's not easily accessible otherwise. Changes to the underlying struct will cause runtime failures in the test case if/when they occur
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
