* add output formatting control to cli
Adds the capacity for the CLI to output either a
text table or a json blob to the terminal. Table is
the default behavior, json is toggled with the --json
flag.
* reduce restore point details to matching refs
A selector should be able to reduce a set of restore point
details to only those that pass its inclusion and
exclusion rules.
Update the backup operation to create RestorePoint and RestorePointDetails models in the repository
Add modelstore to the operation to allow backup/restore operations to update/query for corso models
Closes#268
* wire selectors up through backup handling
Selectors are implemented enough to add them end-
to-end in some places. This starts with backup
creation, since that's the most stable set of code in
the repo at the moment.
* refactor the selector api to scope factories
Rather than have the service selector offer an api of
include* or exclude* for each of its recognized
data scopes, instead expose a set of factory funcs
for each datat scope and a smaller api of functions
that can consume those scopes.
* Split KopiaWrapper into repo handle and logic
With ModelStore, multiple structs need a reference to the kopia repo.
Make a small wrapper class (conn) that can open and initialize a repo. The
wrapper handles concurrent closes and opens and does ref counting to
ensure it only drops the kopia handle when the last reference is closed.
Rename KopiaWrapper to Wrapper and keep backup/restore functionality
in it.
* update selector to match current design
The selector design is progressing in the Showdown doc.
This updates the existing structs to match the expectations in that doc.
* add operation results structs
Operations, both backup and restore, need to hold the
results of their operation, and be able to marshal the struct
to json for output.
* separate tenantID from m365 creds
Now that account.Account is in place, tenant id needs
to get removed from the credential set (it isn't actually
a secret) and placed in the account configuration instead.
* Update how S3 storage structs are generated
* fix bug in printing year of date
* use the name of the test instead of trying to pull name from runtime
* always log the time when making the storage struct
* don't allow user to specify prefix
* Fixup tests for new test storage API
* Update function name and comment
* move config unions to common code
The configuration union handlers in Storage and Account
overlapped significantly in behavior. Moving those helpers into
a common code folder was requested. Although the behavior
was similar across the files, the types were not, requiring
the addition of generics to solve the need.
* hook up restore end-to-end
Now that GC and KW both provide restore operations for a
single message, we can hook up the end-to-end restore
process. Integration tests for this change will follow in the
next PR.
* add /pkg/source skeleton
Source acts as an intermediary between the
client and internal packages (GraphConnector, Kopia) to
specify the scope of data in a Backup or Restore
operation request.
* add Account pattern to mimic storage
Introduces /pkg/account, which produces a generic Account
struct that can be configured from specific provider
details (such as m365 credentials), and serialized back to
those details as needed.
Next steps include replacing repository.Account with the
new account. After that, tenantID should get removed from
the m365 credentials handling and placed into the Account
configuration instead.
Introduces a NewRestore method to the Repository struct.
cli/restore/exchange now generates a restoreOperation
and kicks off a Run() when `corso restore exchange` is called.
The restore operation still only accepts the placeholder
Target string slice.
* read test config from local file
Allows local configuration of the test environment by reading from
a .toml config file. If no file exists, the file read is a no-op. Value
prioritization is specified in the readTestConfig() func.
* use config file for backup create
Now that corso stores its prior connection details in a config
file, that file should get used to retrieve storage details as needed.
credentials requriements surface in many places thorughout corso:
they can be sourced from many locations (envs, files, manually),
and used in many more (cli, repo, kopia). This usage could blossom
into all kinds of duplicate structs sharing similar info. The goal
of this change is to centralize where credentials are declared
and managed, and how they then cascade out to other packages.
* add backup to operations
Extend the operations set with a backupOperation. New backup
operations are created through an existing repository and, once
created, can be Run synchronously.
Integration tests have a skeleton, but the meat of them is on hold
while dependencies undergo implementation.
* remove operation progress
* adds env-based integration test controls
Some integration tests were running locally without being told
to do so via env variables. This adds some controllers for validating
if and when those tests should run, and for ensuring the env
variables necessary to run the test are available.
* testing helper comments, func name change
* add testing timeOf log, skip gc ci tests
* corrected the test skip in GC_test
* update kopia_test with helpers
* rename /testing files to *_test.go
This will ensure these files do not compile into the release binary.
* Revert "rename /testing files to *_test.go"
This reverts commit 04fd2046cc6d11bf3e3eb98d98a3e84b1bba6366.
* introduce operations (#114)
Adds the /internal/operations package. This is a starting point
which only includes the base operation struct and handling.
Per-process (backup/restore) operations to come at a later time.
* adding comments for clarity in op_progress
* Add function to open kopia and keep handle
* Add kopia handle in Repository
* Have Repository manage lifetime of KopiaWrapper
* Have CLI manage lifetime of Repository
* validate required storage props (#85)
Centralizes validation of required storage config properties within
the storage package. Requiremens are checked eagerly at
configuration creation, and lazily at config retrieval.
Additionally, updates /pkg/storage tests to use suites
and assert funcs.
* add validation failure tests to storage
* add common config and encryption passwd
Adds a provider-independent configuration handler, and the
the encryption password config property. The password is used
to encrypt and decrypt the kopia repository properties file.
* fix corso_password in ci.yml
* actually use the corso password in testing
* replace passwd in ci.yml with a secret
* ci.yml secret typo fix
A small collection of changes and code cleanup to successfully run
'corso repo init s3' manually. Adds a default s3 url (might want
this configurable for local testing) and aws session token support.
* add e2e wiring of cli to kopia
Now that pkg/storage and internal/kopia are in place, we
can wire up the init flow from the cli all the way to kopia. Testing
harness for this functionality still needs investigation afterward.
* factor out awsVars struct for s3Cfg
* introduces storage pkg (#25)
Adds a new package /pkg/storage. Storage is used to
communicate storage provider information throughout the corso
app. In particular, it allows per-provider data to cross interface
boundaries without slicing their details.
* use consts for s3 config keys
* introduce repository api scaffold with s3 (#6)
introduces the public repository package in /pkg, with a
skeleton implementation for generating the s3 provider config
and boilerplate for where that usage gets centralized.
* condense init/connect to single action
A double-pass NewRepo() && Connect() allows Repository structs
to exist independent guarantees that their usage will be successful.
This hides the struct generation within the Connect and Initialize
funcs so that the construction and guarantees can occur as
a single process.
