* Make corso respect --version
* Add proper version information when building
* Use unified version string
$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)
With tag: v0.0.0-ceaf04c9
Without tag: unreleased-ceaf04c9
* Add a simple build command to build using Makefile
## Description
This sets up the metrics configs using `ldflags`. Keeping it in draft as I wanted to wait till https://github.com/alcionai/corso/pull/1052 is merged as that will affect how we use it in the CI. I have currently set the base branch to `release-ci` on GH, the branch for https://github.com/alcionai/corso/pull/1052 instead of `main` as the diff would make more sense that way.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1067
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
This makes it so that we have an option to locally run all Makefile targets without having to go through Docker. That said, it retains Docker as the default way to run it, but now we can set an evn variable `CORSO_USE_DOCKER=-1` to skip running through docker. I understand if this looks a bit hacky and don't want to add it in, but thought I wold propose this anyways. While not major, I was able to get a good amount of decrease in build times.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Mostly path manipulations here to allow creating selectors for a backup and paths for the output of a backup given some information about the input. Capable of working for OneDrive and Exchange.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Extends progress bar display with multi-line support.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Adds a check that reviews the load test details.
Details entries to make sure that all users were
involved in the operation, and that for each user,
the test involved all item categories for that app.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Found a few misses in the docs regarding test env setup.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds more info in the build docs for building corso binary locally or using containers. Also improved docs on building corso container image.
@gmatev let me know if I should add anything else specifically to the docs.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes#684
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
adds extensible options to folder-level scopes that allows the caller to specify whether they want a
prefix-comparison matcher or a contains-comparison matcher.
Also corrects the behavior of the prefix filter so that it accurately follows the "target is prefix of input" specification, rather than the reverse.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1133
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Allows listing individual backups, instead of always listing the entire set. Could be expanded to accept multiple backup IDs in the future.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1077
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Use an upload session and chunked writes for large attachments.
This commit moves the logic used for OneDrive file upload to the common `uploadsession` pkg and
leverages that for the upload.
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1115
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Repo `--prefix` values should be normalized with a trailing `/`
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* Fixes#1152
## Test Plan
<!-- How will this be tested prior to merging.-->
- [] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
If any of the dependents were skipped, the CI would skip a job. This resulted in `Docs-Linting` getting skipped because of `Generate-CLI-Docs` job being skipped for doc only change. Now they are both merged into a single job. The internal steps for cli docs generation will still be skipped, but that should not affect the steps below.
*I have not been able to fully test it out as the "file change checker" logic can only be checked after merging it, but it should be working*
Ref: https://github.com/orgs/community/discussions/25224
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Hacks in the bytes written as the iteminfo size during onedrive item restores. Would use the drive itemable, but the size property isn't instantiated locally during creation, and cannot be mutated afterwards.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1113
## Test Plan
- [x] 💪 Manual
* add bool format checks for string flags
Adds a boolean-parseable validator for cli flags
that accept strings representing boolean values.
* fix ctx production in cli integration test
* Move interfaces to common location
Upcoming PRs are using these interfaces across packages. Move them to a
common location so multiple packages can use them without import cycles
etc.
* Allow adding newly created folders to the cache (#1131)
* New function to add folders to cache
Allow adding new folders to the cache. Automatically cache the paths for
the new folders. Also add the new function to the interface.
* Reuse the AddToCache function during population
* Wire up ability to back up a single subfolder of mail (#1132)
* Expand cache to return items in it
Required to allow matching an item's path to a selector as the selector
will not provide which paths it matches on easily.
* Function to get collections from cached folders
Returned collections match any matchers given for the folders
* Thread resolver through iterator functions
Allow the folder resolver to be used in all iterator functions. The
resolver will be tied to the current category and user.
* Choose between using resolver and making queries
Allow either using the resolver to get folders with matching names or
using queries to get them.
* Wire up resolver at entry point
Create a resolver instance for each user/category of data being
backedup.
* Preparation for changing how mail enumeration is done (#1157)
* Step towards redoing mail fetching
Pull out old way to get data into a new function and setup some helper
functions etc.
* Switch to pulling mail items folder by folder (#1158)
* Function to pull mail items given collections
Given a set of collections and IDs for those collections pull the mail
items for each collection.
* Create helper function to fetch mail
New helper function to fetch mail items. This goes through each folder
and gets the items for them individually.
* Wire up new way to fetch mail
Leaves fetch logic for other data types undisturbed.
* Tests for new mail fetching logic
Remove tests that were previously in iterators_test.go and move them to
graph_connector_test.go. These tests only had to do with mail logic.
Tests that handled all data types in iterators_test.go have been updated
to skip mail now.
* Add functions for OneDrive purge
function is setup to return and take purgable instead of just userID,
itemID for deleter.
* Adjust other functions to take purgable
Adjust interface for deleter so that it allows more information to the
delete function. The called function can cast the purgable it was passed
back to the type that was returned by the getter function.
* Move struct definition
Per reviewer comment move struct definition to OneDrive package instead
of leaving in purge command.
* Fix lint error
Likely slipped through while GitHub actions was having issues.
## Description
Wrong service referenced in restore output
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1099
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a new package- Observe- for owning user-
oriented displays like progress bars. This PR adds
an initial progress bar to onedrive backups as a
proof-of-concept. The API is more important than
the specific progress bar package at this time.
Future changes may opt for a different pkg.
Display format currently looks like:
```
59% [=============> ] (6.9/12 kB, 14 MB/s) | Item_Name.txt
```
Known Issues:
* the `progressbar` package does not support multiline output, and [the author is not planning to add support](https://github.com/schollz/progressbar/issues/6). This causes concurrent items to overwrite each other. We will either need to fork the library, or change to a different one.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Some paths can generate identical unescaped strings even though they have different elements. Add a test ensuring they generate unique ShortRefs even under these conditions.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1092
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context. The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton. But in order for the logs to show up
within the test, they need to get flushed.
See my comments for focus on non-chaff changes.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Contact Iterative sequence tests changed to reflect `exchange.Backup` functions used within `SetupExchangeCollectionVars()`. The previous bug was caused by misalignment from the old test configuration and the current implementation.
<!-- Insert PR description-->
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #1107<issue>
## Test Plan
- [x] ⚡ Unit test
* Setup CI for release
* Initial commit of CHANGELOG.md
* Pin vale and markdownlint-cli to current versions
* Update branch name for CI
* Pull request workflow job
## Description
Useful helper functions for the purge script
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #1090
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
CI test timing during times of high activity. Reduced test suite to cover Inbox and adjusted outputs accordingly.
<!-- Insert PR description-->
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #1078<issue>
## Test Plan
- [x] ⚡ Unit test
## Description
Updated permissions for Azure AD App
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Fixes#1060
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [] 💚 E2E
## Description
* Fill in developer/build.md page placeholder
* Fix some liniting issues
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds validation of cli inputs for time-based flags.
All time values that can be parsed by the time
handling in common count as supported, even
though official support includes a smaller set.
Also adds some clean-up to time.go and adds a
design document describing standard time value
formats, and the maintenance thereof, in corso.
## Type of change
- [x] 🌻 Feature
- [x] 🗺️ Documentation
- [x] 🤖 Test
## Issue(s)
* #943
* #924
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
purge restored folders for all users in the tenant following a load test to ensure we don't over-
produce data that causes future tests to time out.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #1048
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Currently only the ModelStore populates and checks versions.
Some high-level points:
* fail any sort of get (just metadata or full model) if there's a version mismatch
* update operations automatically set things to the current version
* versions are stored as tags so there's some int->string (and vice versa) munging
* versions stored as tags so they can be repopulated even if only the metadata (BaseModel) for a model is pulled. This is done mostly for consistency
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#284
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Removes the -all flag from the exchange cli. The
same functionality can be gained with the flag
--user *.
Additionally does some minor tidying of other
comments and outputs throughout the cli.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1032
## Test Plan
- [x] ⚡ Unit test
## Description
Load Testing is causing folder growth (fix is underway)
and is already taking > 4 hours only to back-up emails.
This reduces the dataset to only the primary folders for
each exchange type. We can expand from here when
data production is better under control.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Test to check that attachments are properly populated and fetched during restore and backup
Some notes:
* make slice element comparison function generic so it can compare locations and attachments
* very hacky comparison for attachment content using reflection as it's not easily accessible otherwise. Changes to the underlying struct will cause runtime failures in the test case if/when they occur
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
The feature allows for the subFolders to be created. Test Suite Expanded.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*Closes Issue #1054<issue>
## Test Plan
- [x] ⚡ Unit test
