* Skip updating policy if no changes needed
* Set default compressor on repo connection and initialization
* Factor out common code for connecting to a repo
* New unit test ensuring compressor is set properly
Test compressor is set both through Initialize and Connect because
client code doesn't have to call Connect if it just did Initialize.
* Function that allows adjusting global compression
Takes the name of a kopia compressor and sets the default compressor to
be that in the repo.
* Basic unit tests for setting compression
Ensures that the policy is actually applied to future snapshots that
kopia will take.
* Integration test for changing compression types
Make sure that data is still available even if the global compression
policy changes from what it was when the backup was made.
* Enable line width linter
Set to 120 which should be long enough to not be annoying but keep
things from getting "too long." Adding to get rid of the subjectiveness
of what is "too long." Tabs count as a single character.
* Remove artificial limit on kopia directories
Originally did not allow a directory to have both child directories and
items. Remove that limit and move logic to execute callbacks on static
items to the iteration function.
* Update tests for new kopia directory structure
* Turn on revive linter, ignore only a few things
* Fix lint errors
Ignore shadowing of 'suite' in tests for now. Also move some constants
that had the same value to tester.
Kopia maintainer mentioned this was a better way to deal with it than
the old version. Mostly superficial changes to logic.
Always safe to flush the write session during an upload. That will
ensure write sessions are closed. Validity of the uploaded data will
be via reachability of the data in the repo.
Drawback is if there's a problem flushing the write session it will
return that error instead of the original error (if there was one)
The alias of testing to ctesting prevented auto-importing
of the testing library. This change (arbitrarily) renames
the package so that it doesn't collide with the core pkg
for "testing".
embedding a baseModel means that objects like backup
have a backup.StableID, but no backup.ID nor backup.
backupID. This could be confusing for users. This change
swaps the value and type names, so that baseModels
include an ID value of type StableID.
* store backup operation results in the backup manifest
Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry. These additional details will appear when users
call `corso backup list`.
* [Minor #323] add --success-on-exists flag to make idempotent
Co-authored-by: ryanfkeepers <ryanfkeepers@gmail.com>
Co-authored-by: Keepers <104464746+ryanfkeepers@users.noreply.github.com>
* adds store package for wrapping model_store
Introduces the pkg/store package, which contains funcs
for wrapping the model_store with common requests.
This package choice was made for its combination
of being in an accessible place, centralizing functionality
and not introducing circular dependencies.
* add e2e backup-restore integration test
Adds an e2e integration test that starts by backing up
data, and ends with restoring it. Also makes various
amendments to other code where necessary to
facilitate this exercise.
These fields can be used to lookup snapshots in kopia as they are part
of the `Source` struct. They are also stored in the kopia sessions but
they should not cause session collisions. Sessions have unique IDs to go
along with this information.
* Implement getting data for directory subtree
Return a slice of collections with data for a given directory subtree in
kopia. Traverse the full directory before creating a DataCollection
instead of sending items as they are found because future
implementations may cause blocking on send. This could reduce
parallelism because the code won't be able to find other directories to
traverse until the files are seen. Kopia also currently loads the entire
directory at once so there's not much benefit to streaming.
System will now continuing pulling data until completion and report all
errors at the end of the run.
* Tests for getting persisted subtree data including some error cases
Update the backup operation to create RestorePoint and RestorePointDetails models in the repository
Add modelstore to the operation to allow backup/restore operations to update/query for corso models
Closes#268
* Factor out common code for getting kopia items
Both directory and single item restore in kopia need to do common tasks
like getting the item in question. Factor out that common code and
adjust tests to prep for directory restore.
Use a slice to back the data instead of adding directly to the channel
for two reasons (this may change in the future though):
* kopia loads all data about a directory at the same time
* consumers of the DataCollection may not pull items from the channel
at a fast rate, which could block adding to the channel. This could
lead to delays in discovering other directories to traverse in
multi-threaded scenarios
* Split KopiaWrapper into repo handle and logic
With ModelStore, multiple structs need a reference to the kopia repo.
Make a small wrapper class (conn) that can open and initialize a repo. The
wrapper handles concurrent closes and opens and does ref counting to
ensure it only drops the kopia handle when the last reference is closed.
Rename KopiaWrapper to Wrapper and keep backup/restore functionality
in it.
Tests that run multiple sub-tests do not use the fields in the test
suite because that would cause the model store instance to be reused
instead of having a new model store instance for each subtest.
* Implement ModelStore GetByType and Get
* Add tests for ModelStore Get functions
* Add stricter "type" checks for loaded models
Take modelType as parameter and check the model in question matches that
type. Adds a little extra layer of protection if models happen to have
the same struct layout.
* Add ModelStore Update operation
* Tests for ModelStore Update function
* Add regression test for error during Update()
Ensure that if an error occurs during a ModelStore update operation the
previously stored model remains unchanged and no new model is visible to
the user.
* Simple Get and Put implementations
Get implementation is currently the one that uses the kopia ID of the
model/manifest.
* Basic tests for ModelStore Get/Put
* Change DataCollection to return channel directly
Precursor to restoring multiple items from kopia. Allows one to keep a
DataCollection open until all items are processed without blocking
consumers of the DataCollection (they can use a select-block if needed).
* Update tests for new DataCollection interface
* Handle context cancellation with DataCollection
* separate tenantID from m365 creds
Now that account.Account is in place, tenant id needs
to get removed from the credential set (it isn't actually
a secret) and placed in the account configuration instead.
* Update how S3 storage structs are generated
* fix bug in printing year of date
* use the name of the test instead of trying to pull name from runtime
* always log the time when making the storage struct
* don't allow user to specify prefix
* Fixup tests for new test storage API
* Update function name and comment
* hook up restore end-to-end
Now that GC and KW both provide restore operations for a
single message, we can hook up the end-to-end restore
process. Integration tests for this change will follow in the
next PR.
* DataCollection and DataStream structs for kopia
* Helper function to get a single item of data
Returns a DataCollection with a single item.
* Test for backing up and restoring a single item
* read test config from local file
Allows local configuration of the test environment by reading from
a .toml config file. If no file exists, the file read is a no-op. Value
prioritization is specified in the readTestConfig() func.
Fill in BackupCollection logic for kopia dirs and test
"Rehydrate" the given DataCollections into a hierarchy of kopia
virtualfs directories, allowing kopia to make a single snapshot of them.
Directory structure is extracted from the information returned by
DataCollection.FullPath().
Add skeleton and test for backing up data with kopia
Full code for launching kopia snapshots is present but code for building
the directory structure to hand kopia is not. Therefore, no data will be
backed up right now.
