* Enable line width linter
Set to 120 which should be long enough to not be annoying but keep
things from getting "too long." Adding to get rid of the subjectiveness
of what is "too long." Tabs count as a single character.
* Remove artificial limit on kopia directories
Originally did not allow a directory to have both child directories and
items. Remove that limit and move logic to execute callbacks on static
items to the iteration function.
* Update tests for new kopia directory structure
* Turn on revive linter, ignore only a few things
* Fix lint errors
Ignore shadowing of 'suite' in tests for now. Also move some constants
that had the same value to tester.
Kopia maintainer mentioned this was a better way to deal with it than
the old version. Mostly superficial changes to logic.
Always safe to flush the write session during an upload. That will
ensure write sessions are closed. Validity of the uploaded data will
be via reachability of the data in the repo.
Drawback is if there's a problem flushing the write session it will
return that error instead of the original error (if there was one)
* store backup operation results in the backup manifest
Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry. These additional details will appear when users
call `corso backup list`.
* add e2e backup-restore integration test
Adds an e2e integration test that starts by backing up
data, and ends with restoring it. Also makes various
amendments to other code where necessary to
facilitate this exercise.
These fields can be used to lookup snapshots in kopia as they are part
of the `Source` struct. They are also stored in the kopia sessions but
they should not cause session collisions. Sessions have unique IDs to go
along with this information.
* Implement getting data for directory subtree
Return a slice of collections with data for a given directory subtree in
kopia. Traverse the full directory before creating a DataCollection
instead of sending items as they are found because future
implementations may cause blocking on send. This could reduce
parallelism because the code won't be able to find other directories to
traverse until the files are seen. Kopia also currently loads the entire
directory at once so there's not much benefit to streaming.
System will now continuing pulling data until completion and report all
errors at the end of the run.
* Tests for getting persisted subtree data including some error cases
Update the backup operation to create RestorePoint and RestorePointDetails models in the repository
Add modelstore to the operation to allow backup/restore operations to update/query for corso models
Closes#268
* Factor out common code for getting kopia items
Both directory and single item restore in kopia need to do common tasks
like getting the item in question. Factor out that common code and
adjust tests to prep for directory restore.
Use a slice to back the data instead of adding directly to the channel
for two reasons (this may change in the future though):
* kopia loads all data about a directory at the same time
* consumers of the DataCollection may not pull items from the channel
at a fast rate, which could block adding to the channel. This could
lead to delays in discovering other directories to traverse in
multi-threaded scenarios
* Split KopiaWrapper into repo handle and logic
With ModelStore, multiple structs need a reference to the kopia repo.
Make a small wrapper class (conn) that can open and initialize a repo. The
wrapper handles concurrent closes and opens and does ref counting to
ensure it only drops the kopia handle when the last reference is closed.
Rename KopiaWrapper to Wrapper and keep backup/restore functionality
in it.
