## Description
Introduces a OneDrive data collection.
Follow up PRs will implement the `collection.driveItemReader()` method that uses the Graph API
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
- #387
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
* Enable line width linter
Set to 120 which should be long enough to not be annoying but keep
things from getting "too long." Adding to get rid of the subjectiveness
of what is "too long." Tabs count as a single character.
* Turn on revive linter, ignore only a few things
* Fix lint errors
Ignore shadowing of 'suite' in tests for now. Also move some constants
that had the same value to tester.
* Add import order linter
Error messages aren't the most clear/helpful, but they will point you to
the lines that need modifying.
* Adjust max errors reported
* Fix lint errors
The alias of testing to ctesting prevented auto-importing
of the testing library. This change (arbitrarily) renames
the package so that it doesn't collide with the core pkg
for "testing".
In order for sdk users to build an operation using a repo,
without importing /inernal packages, the operation
options needs to be available via a /pkg package.
* add jsonDebug, MinimumPrintable to interface
We want to optionally print print either debugging
(containing all internal data) or user-friendly json
outputs. This requires a new flag: json-debug, and
an extension to the Print interface: minimumPrintable.
embedding a baseModel means that objects like backup
have a backup.StableID, but no backup.ID nor backup.
backupID. This could be confusing for users. This change
swaps the value and type names, so that baseModels
include an ID value of type StableID.
* store backup operation results in the backup manifest
Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry. These additional details will appear when users
call `corso backup list`.
* reduce filter selector flags to single strings
Since it doesn't make sense for many of the filter-type
selector flags to receive multiple input values (ex: what
would --received-after date1,date2 result in?), the filter
flags are getting reduced to single-string values.
* add must-pass filters to selectors
Extends the selectors scope set to include Filters. This
allows users to define all-pass matchers (filters),
separate from any-pass matchers (includes), and global
exclusions.
When a caller constructs an Users() exchange scope,
instead of building a single scope that contains all of the
provided users and all of the data beneath, follow the
pattern of the other scopes and generate one scope per
user-category pair.
* expand restore flags
Adds support for info filter flags to the restore command.
Also adds (hidden) support for event and contact data
types. Finally, exchnages single-string flag data types for
multi-value string array support.
* filter backup details by flags
`backup details` should have its output filtered by the flags provided by
the user. In addition, the selector's FilterDetails should maintain
information (esp service info) about the entries, rather than slicing them
down to only the path reference.
* filter backup details by flags
`backup details` should have its output filtered by the flags provided by
the user. In addition, the selector's FilterDetails should maintain
information (esp service info) about the entries, rather than slicing them
down to only the path reference.
* Basic golangci-lint config with gofmt and timeout
Remove corresponding sections from other CI config files.
* Enable comment and string spelling linter
* Fix spelling mistakes for linter
Co-authored-by: Danny <danny@alcion.ai>
* introduce exchange info selector support
Adds support in selectors/exchange for queries based on
backup.ExchangeInfo entries. This allows the declaration
of selectors based on non-identifier details such as sender,
subject, or receivedAt time.
Changes Exclude scope matching from being an Any-
match comparator to an All-match. This keeps exclude
and include behavior identical, hopefully making less
confusion for users.
* filter backup details by flags
`backup details` should have its output filtered by the flags provided by
the user. In addition, the selector's FilterDetails should maintain
information (esp service info) about the entries, rather than slicing them
down to only the path reference.
This allows the user to specify repo configuration via environment variables when a config file is not available.
This is a temporary fix - ideally we would leverage Viper for this but we currently only use Viper if a config file
is available so that needs to be refactored a bit.
[1]
`TENANT_ID` (already supported)
`BUCKET`
`ENDPOINT`
`PREFIX`
* refactor selector scopes to accept slices
Cli flag implementation was showcasing a toil issue: building
selectors required a lot of repetitious code for combining
inputs into sets of scopes. Since all of these productions
were effectively identical (eg: for each user, then each folder,
create a scope with the ids), the cleaner solution is to pack
that behavior into the scope constructors themselves.
* adds the onError option to operations
Adds the OnError option to operations.Options. OnError tells
corso whether to continue despite concurrent processing
errors, or to exit processing on any error. Also includes flag
support for setting the option. Only adds the options, does
not assert error handling behavior in corso.
* adds store package for wrapping model_store
Introduces the pkg/store package, which contains funcs
for wrapping the model_store with common requests.
This package choice was made for its combination
of being in an accessible place, centralizing functionality
and not introducing circular dependencies.
* add e2e backup-restore integration test
Adds an e2e integration test that starts by backing up
data, and ends with restoring it. Also makes various
amendments to other code where necessary to
facilitate this exercise.
* add output formatting control to cli
Adds the capacity for the CLI to output either a
text table or a json blob to the terminal. Table is
the default behavior, json is toggled with the --json
flag.
* reduce restore point details to matching refs
A selector should be able to reduce a set of restore point
details to only those that pass its inclusion and
exclusion rules.
Update the backup operation to create RestorePoint and RestorePointDetails models in the repository
Add modelstore to the operation to allow backup/restore operations to update/query for corso models
Closes#268
* wire selectors up through backup handling
Selectors are implemented enough to add them end-
to-end in some places. This starts with backup
creation, since that's the most stable set of code in
the repo at the moment.
* refactor the selector api to scope factories
Rather than have the service selector offer an api of
include* or exclude* for each of its recognized
data scopes, instead expose a set of factory funcs
for each datat scope and a smaller api of functions
that can consume those scopes.
* Split KopiaWrapper into repo handle and logic
With ModelStore, multiple structs need a reference to the kopia repo.
Make a small wrapper class (conn) that can open and initialize a repo. The
wrapper handles concurrent closes and opens and does ref counting to
ensure it only drops the kopia handle when the last reference is closed.
Rename KopiaWrapper to Wrapper and keep backup/restore functionality
in it.
* update selector to match current design
The selector design is progressing in the Showdown doc.
This updates the existing structs to match the expectations in that doc.
