Makes the necessary changes, including adding helper
funcs, to bring the CLI up to an integration-testable
state. The changes made in this commit should be
sufficient for most other CLI tests. Includes a single
test as verification.
* Add import order linter
Error messages aren't the most clear/helpful, but they will point you to
the lines that need modifying.
* Adjust max errors reported
* Fix lint errors
The alias of testing to ctesting prevented auto-importing
of the testing library. This change (arbitrarily) renames
the package so that it doesn't collide with the core pkg
for "testing".
Cobra's RunE causes every error response to display the
CLI help menu for the current command, after printing the
original error itself. This made error outputs difficult to
read, and was generally unhelpful in most cases. The exit
func in Print now prints the error and closes the CLI without
showing the help menu.
In order for sdk users to build an operation using a repo,
without importing /inernal packages, the operation
options needs to be available via a /pkg package.
* return errors if selectors reduce all entries
During a `backup details` or `restore` command, if the
selectors produce 0 results, return an error rather than
running a no-op process.
* add jsonDebug, MinimumPrintable to interface
We want to optionally print print either debugging
(containing all internal data) or user-friendly json
outputs. This requires a new flag: json-debug, and
an extension to the Print interface: minimumPrintable.
* store backup operation results in the backup manifest
Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry. These additional details will appear when users
call `corso backup list`.
* reduce filter selector flags to single strings
Since it doesn't make sense for many of the filter-type
selector flags to receive multiple input values (ex: what
would --received-after date1,date2 result in?), the filter
flags are getting reduced to single-string values.
* add must-pass filters to selectors
Extends the selectors scope set to include Filters. This
allows users to define all-pass matchers (filters),
separate from any-pass matchers (includes), and global
exclusions.
When a caller constructs an Users() exchange scope,
instead of building a single scope that contains all of the
provided users and all of the data beneath, follow the
pattern of the other scopes and generate one scope per
user-category pair.
* [Minor #323] add --success-on-exists flag to make idempotent
Co-authored-by: ryanfkeepers <ryanfkeepers@gmail.com>
Co-authored-by: Keepers <104464746+ryanfkeepers@users.noreply.github.com>
* expand restore flags
Adds support for info filter flags to the restore command.
Also adds (hidden) support for event and contact data
types. Finally, exchnages single-string flag data types for
multi-value string array support.
* filter backup details by flags
`backup details` should have its output filtered by the flags provided by
the user. In addition, the selector's FilterDetails should maintain
information (esp service info) about the entries, rather than slicing them
down to only the path reference.
* Basic golangci-lint config with gofmt and timeout
Remove corresponding sections from other CI config files.
* Enable comment and string spelling linter
* Fix spelling mistakes for linter
Co-authored-by: Danny <danny@alcion.ai>
* introduce exchange info selector support
Adds support in selectors/exchange for queries based on
backup.ExchangeInfo entries. This allows the declaration
of selectors based on non-identifier details such as sender,
subject, or receivedAt time.
Changes Exclude scope matching from being an Any-
match comparator to an All-match. This keeps exclude
and include behavior identical, hopefully making less
confusion for users.
* filter backup details by flags
`backup details` should have its output filtered by the flags provided by
the user. In addition, the selector's FilterDetails should maintain
information (esp service info) about the entries, rather than slicing them
down to only the path reference.
This allows the user to specify repo configuration via environment variables when a config file is not available.
This is a temporary fix - ideally we would leverage Viper for this but we currently only use Viper if a config file
is available so that needs to be refactored a bit.
[1]
`TENANT_ID` (already supported)
`BUCKET`
`ENDPOINT`
`PREFIX`
* add per-data type flags to backup details
Adds the contacts, contact folders, emails, email folders, events,
and users flag support to `corso backup create exchange`
to enable selector support.
* refactor selector scopes to accept slices
Cli flag implementation was showcasing a toil issue: building
selectors required a lot of repetitious code for combining
inputs into sets of scopes. Since all of these productions
were effectively identical (eg: for each user, then each folder,
create a scope with the ids), the cleaner solution is to pack
that behavior into the scope constructors themselves.
* adds the onError option to operations
Adds the OnError option to operations.Options. OnError tells
corso whether to continue despite concurrent processing
errors, or to exit processing on any error. Also includes flag
support for setting the option. Only adds the options, does
not assert error handling behavior in corso.
* adds store package for wrapping model_store
Introduces the pkg/store package, which contains funcs
for wrapping the model_store with common requests.
This package choice was made for its combination
of being in an accessible place, centralizing functionality
and not introducing circular dependencies.
* add --all and --data to backup create
Adds flags for backing up all exchange data, and
for isolating the data in the backup by data type.
Introduces validation and selector creation in
backup create. Switches the --user flag variable
type from a string to a []string.
* add e2e backup-restore integration test
Adds an e2e integration test that starts by backing up
data, and ends with restoring it. Also makes various
amendments to other code where necessary to
facilitate this exercise.
* add output formatting control to cli
Adds the capacity for the CLI to output either a
text table or a json blob to the terminal. Table is
the default behavior, json is toggled with the --json
flag.
* wire selectors up through backup handling
Selectors are implemented enough to add them end-
to-end in some places. This starts with backup
creation, since that's the most stable set of code in
the repo at the moment.
* allow connect to create .corso config file
Current bug: if no .corso config file exists, then repo connect
will always fail, even if it has the correct details to build
a new config file. Solution: allow connect to build a .corso
config file when missing, so long as the operation succeeds
otherwise.
In tandem, return an error whenever a user attempts to
call repo connect with details that do not match the existing
.corso config file.
* Add the backup list command to the cli
Adds `corso backup list <service>` to the cli commands.
Currently does nothing, while we're waiting on
downstream implementation. e2e hookup will arrive later.
* separate tenantID from m365 creds
Now that account.Account is in place, tenant id needs
to get removed from the credential set (it isn't actually
a secret) and placed in the account configuration instead.
* hook up restore end-to-end
Now that GC and KW both provide restore operations for a
single message, we can hook up the end-to-end restore
process. Integration tests for this change will follow in the
next PR.
Introduces a NewRestore method to the Repository struct.
cli/restore/exchange now generates a restoreOperation
and kicks off a Run() when `corso restore exchange` is called.
The restore operation still only accepts the placeholder
Target string slice.
Adds the `corso restore` command to the cli (which only displays
a help menu). And adds the `corso restore exchange`
command (which also only displays a help menu) to the cli.
Further functionality (like the complete flags) will come later.
