## Description
migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.
⚠️ Your local environment will need to change in the following ways: ⚠️
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #558
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Exposing user IDs in addition to user emails.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [X] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [X] ⚡ Unit test
- [ ] 💚 E2E
## Description
Use paths package to trim unescaped trailing '/' characters from input for Exchange mail and OneDrive folder names. Add tests for Exchange showing that the trimming also works properly if the folder name ends with '/'.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1147
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Watch the minimum-complete load test over the weekend.
Backup only, all tenant users, all data types, no restrictions.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Adds scripted production of mock emails for building out
large/rolling datasets for load testing.
## Type of change
- [x] 🌻 Feature
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
## Description
Added an additional set of test suites to load_test
to focus on the backup of a single user who will
have their account prepared with a significant amount
of data (many emails, many events, large files, etc).
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Introduces a `factory` script to help devs generate
fake m365 data in bulk for testing purposes. Only
boilerplate at this time. Any TODO's or not-yet-
implemented statuses are present as placeholders
alone, without official deadlines.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
Adds a processor that confirms whether user has
added a value for a flag in the cmd, or if it is the
default value. This map of valued flags is added
to the service opts structs to for validation.
Also migrates many service flags to utils as
consts so that these values can be maintained
as consistent across packages.
## Description
Feature to add the folder hierarchy for folders when restored.
This required an overhaul of the `graph.ContainerResolver` interfaces:
- MailFolderCache
- ContactFolderCache
- ~EventFolderCache (placed in a separate PR)~ https://github.com/alcionai/corso/pull/1101
Restore Pipeline changed to separate the caching / container creation process from the rest of the restore pipeline.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
* closes#1046
* #1004
* closes#1091
* closes#1098
* closes#1097
* closes#1096
* closes#1095
* closes#991
* closes#895
* closes#798
## Test Plan
- [x] ⚡ Unit test
* Make corso respect --version
* Add proper version information when building
* Use unified version string
$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)
With tag: v0.0.0-ceaf04c9
Without tag: unreleased-ceaf04c9
* Add a simple build command to build using Makefile
## Description
This sets up the metrics configs using `ldflags`. Keeping it in draft as I wanted to wait till https://github.com/alcionai/corso/pull/1052 is merged as that will affect how we use it in the CI. I have currently set the base branch to `release-ci` on GH, the branch for https://github.com/alcionai/corso/pull/1052 instead of `main` as the diff would make more sense that way.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1067
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
This makes it so that we have an option to locally run all Makefile targets without having to go through Docker. That said, it retains Docker as the default way to run it, but now we can set an evn variable `CORSO_USE_DOCKER=-1` to skip running through docker. I understand if this looks a bit hacky and don't want to add it in, but thought I wold propose this anyways. While not major, I was able to get a good amount of decrease in build times.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Mostly path manipulations here to allow creating selectors for a backup and paths for the output of a backup given some information about the input. Capable of working for OneDrive and Exchange.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Extends progress bar display with multi-line support.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Adds a check that reviews the load test details.
Details entries to make sure that all users were
involved in the operation, and that for each user,
the test involved all item categories for that app.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Found a few misses in the docs regarding test env setup.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds more info in the build docs for building corso binary locally or using containers. Also improved docs on building corso container image.
@gmatev let me know if I should add anything else specifically to the docs.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes#684
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
adds extensible options to folder-level scopes that allows the caller to specify whether they want a
prefix-comparison matcher or a contains-comparison matcher.
Also corrects the behavior of the prefix filter so that it accurately follows the "target is prefix of input" specification, rather than the reverse.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1133
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Allows listing individual backups, instead of always listing the entire set. Could be expanded to accept multiple backup IDs in the future.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1077
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Use an upload session and chunked writes for large attachments.
This commit moves the logic used for OneDrive file upload to the common `uploadsession` pkg and
leverages that for the upload.
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1115
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Repo `--prefix` values should be normalized with a trailing `/`
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* Fixes#1152
## Test Plan
<!-- How will this be tested prior to merging.-->
- [] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
If any of the dependents were skipped, the CI would skip a job. This resulted in `Docs-Linting` getting skipped because of `Generate-CLI-Docs` job being skipped for doc only change. Now they are both merged into a single job. The internal steps for cli docs generation will still be skipped, but that should not affect the steps below.
*I have not been able to fully test it out as the "file change checker" logic can only be checked after merging it, but it should be working*
Ref: https://github.com/orgs/community/discussions/25224
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Hacks in the bytes written as the iteminfo size during onedrive item restores. Would use the drive itemable, but the size property isn't instantiated locally during creation, and cannot be mutated afterwards.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1113
## Test Plan
- [x] 💪 Manual
* add bool format checks for string flags
Adds a boolean-parseable validator for cli flags
that accept strings representing boolean values.
* fix ctx production in cli integration test
* Move interfaces to common location
Upcoming PRs are using these interfaces across packages. Move them to a
common location so multiple packages can use them without import cycles
etc.
* Allow adding newly created folders to the cache (#1131)
* New function to add folders to cache
Allow adding new folders to the cache. Automatically cache the paths for
the new folders. Also add the new function to the interface.
* Reuse the AddToCache function during population
* Wire up ability to back up a single subfolder of mail (#1132)
* Expand cache to return items in it
Required to allow matching an item's path to a selector as the selector
will not provide which paths it matches on easily.
* Function to get collections from cached folders
Returned collections match any matchers given for the folders
* Thread resolver through iterator functions
Allow the folder resolver to be used in all iterator functions. The
resolver will be tied to the current category and user.
* Choose between using resolver and making queries
Allow either using the resolver to get folders with matching names or
using queries to get them.
* Wire up resolver at entry point
Create a resolver instance for each user/category of data being
backedup.
* Preparation for changing how mail enumeration is done (#1157)
* Step towards redoing mail fetching
Pull out old way to get data into a new function and setup some helper
functions etc.
* Switch to pulling mail items folder by folder (#1158)
* Function to pull mail items given collections
Given a set of collections and IDs for those collections pull the mail
items for each collection.
* Create helper function to fetch mail
New helper function to fetch mail items. This goes through each folder
and gets the items for them individually.
* Wire up new way to fetch mail
Leaves fetch logic for other data types undisturbed.
* Tests for new mail fetching logic
Remove tests that were previously in iterators_test.go and move them to
graph_connector_test.go. These tests only had to do with mail logic.
Tests that handled all data types in iterators_test.go have been updated
to skip mail now.
* Add functions for OneDrive purge
function is setup to return and take purgable instead of just userID,
itemID for deleter.
* Adjust other functions to take purgable
Adjust interface for deleter so that it allows more information to the
delete function. The called function can cast the purgable it was passed
back to the type that was returned by the getter function.
* Move struct definition
Per reviewer comment move struct definition to OneDrive package instead
of leaving in purge command.
* Fix lint error
Likely slipped through while GitHub actions was having issues.
## Description
Wrong service referenced in restore output
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1099
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a new package- Observe- for owning user-
oriented displays like progress bars. This PR adds
an initial progress bar to onedrive backups as a
proof-of-concept. The API is more important than
the specific progress bar package at this time.
Future changes may opt for a different pkg.
Display format currently looks like:
```
59% [=============> ] (6.9/12 kB, 14 MB/s) | Item_Name.txt
```
Known Issues:
* the `progressbar` package does not support multiline output, and [the author is not planning to add support](https://github.com/schollz/progressbar/issues/6). This causes concurrent items to overwrite each other. We will either need to fork the library, or change to a different one.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Some paths can generate identical unescaped strings even though they have different elements. Add a test ensuring they generate unique ShortRefs even under these conditions.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1092
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context. The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton. But in order for the logs to show up
within the test, they need to get flushed.
See my comments for focus on non-chaff changes.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Contact Iterative sequence tests changed to reflect `exchange.Backup` functions used within `SetupExchangeCollectionVars()`. The previous bug was caused by misalignment from the old test configuration and the current implementation.
<!-- Insert PR description-->
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1107<issue>
## Test Plan
- [x] ⚡ Unit test
* Setup CI for release
* Initial commit of CHANGELOG.md
* Pin vale and markdownlint-cli to current versions
* Update branch name for CI
* Pull request workflow job
## Description
Useful helper functions for the purge script
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1090
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
CI test timing during times of high activity. Reduced test suite to cover Inbox and adjusted outputs accordingly.
<!-- Insert PR description-->
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1078<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Updated permissions for Azure AD App
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Fixes#1060
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [] 💚 E2E
## Description
* Fill in developer/build.md page placeholder
* Fix some liniting issues
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds validation of cli inputs for time-based flags.
All time values that can be parsed by the time
handling in common count as supported, even
though official support includes a smaller set.
Also adds some clean-up to time.go and adds a
design document describing standard time value
formats, and the maintenance thereof, in corso.
## Type of change
- [x] 🌻 Feature
- [x] 🗺️ Documentation
- [x] 🤖 Test
## Issue(s)
* #943
* #924
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
