## Description
Other components may need to rebuild the directory hierarchy of items. As the paths Corso deals with can be hard to properly parse at times, store that information in the Corso backup details. The hierarchy can be rebuilt by following the `ParentRef` fields of items. The item at the root of the hierarchy has an empty `ParentRef` field.
Also hide these folders from end-users. They are not displayed during backup list nor are they eligible as a target for restore
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#862
* closes#861
* closes#818
merge after:
* #869
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Short refs allow the user to type less to select an item to restore. This PR
* Adds a short ref field to the BackupDetails entries
* Populates the short ref field during backup
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #572
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
* Make data.Collection.FullPath return path.Path
* Fixup graph connector code for path struct
* Add Elements call to Path interface
Still should not be used in place of the named functions to get specific
path elements (e.x. ResourceOwner())
* Fixup kopia wrapper path handling
Mostly removing shim code and minor fixup for building the directory
tree.
* All the test fixes
* Helper function to append elements to a path
Kopia wrapper will need to create the complete path of an item by
joining the collection path and the item name. This allows it to do so
without having to drop to a path Builder (not service/category safe) and
go back to a resource path. Right now it does not handle escaping.
* Use path struct while streaming entries
Use new path struct while streaming entries to kopia. Preparation for
FullPath returning a path struct.
* Update tests to use valid path structures
* Use path struct in kopia DataCollection
Does not change the external API of the DataCollection any, just updates
internals in preparation for switching support of
data.Collection.FullPath.
* Expand Path interface slightly
kopia.Wrapper needs some extra functionality from paths, mostly along
the lines of directly manipulating the elements in the path. This gives
access to those functions.
* Use path struct in kopia.Wrapper for restore
Pass path structs to the newly created collections during restore.
* Add tests for new path functionality
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.
All other changes in the repository update the
package imports to the new module path.
Introduces manual deletion of existing backups. The delete
includes: the modelStore backup, modelStore details, and
the kopia snapshot of the backup itself.
* Update to kopia with required callback
* Support structs for materializing backup details
Kopia will not allow us to pass data to it that should be passed back to
us in the `FinishedFile` callback. To work around this, create a small
thread-safe support struct that handles information about files kopia is
currently processing. Entries are removed from the set when kopia is
done with them and if no error occurred, the item's info will be added
to the BackupDetails.
* Switch to best attempt for iterating through files
Defaulting to "best-attempt" error handling where all data that didn't
result in an error is handed to kopia and then all errors encountered
are returned at the end.
* Test for uploads that have an error
Simple error reading a file. BackupDetails should not contain
information about the file that had the error (needs update to kopia
code and this code to pass). All other files should be present in kopia
and in BackupDetails.
Co-authored-by: Danny <danny@alcion.ai>
* Enable line width linter
Set to 120 which should be long enough to not be annoying but keep
things from getting "too long." Adding to get rid of the subjectiveness
of what is "too long." Tabs count as a single character.
* Remove artificial limit on kopia directories
Originally did not allow a directory to have both child directories and
items. Remove that limit and move logic to execute callbacks on static
items to the iteration function.
* Update tests for new kopia directory structure
* Turn on revive linter, ignore only a few things
* Fix lint errors
Ignore shadowing of 'suite' in tests for now. Also move some constants
that had the same value to tester.
Kopia maintainer mentioned this was a better way to deal with it than
the old version. Mostly superficial changes to logic.
Always safe to flush the write session during an upload. That will
ensure write sessions are closed. Validity of the uploaded data will
be via reachability of the data in the repo.
Drawback is if there's a problem flushing the write session it will
return that error instead of the original error (if there was one)
* store backup operation results in the backup manifest
Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry. These additional details will appear when users
call `corso backup list`.
* add e2e backup-restore integration test
Adds an e2e integration test that starts by backing up
data, and ends with restoring it. Also makes various
amendments to other code where necessary to
facilitate this exercise.
These fields can be used to lookup snapshots in kopia as they are part
of the `Source` struct. They are also stored in the kopia sessions but
they should not cause session collisions. Sessions have unique IDs to go
along with this information.
* Implement getting data for directory subtree
Return a slice of collections with data for a given directory subtree in
kopia. Traverse the full directory before creating a DataCollection
instead of sending items as they are found because future
implementations may cause blocking on send. This could reduce
parallelism because the code won't be able to find other directories to
traverse until the files are seen. Kopia also currently loads the entire
directory at once so there's not much benefit to streaming.
System will now continuing pulling data until completion and report all
errors at the end of the run.
* Tests for getting persisted subtree data including some error cases
Update the backup operation to create RestorePoint and RestorePointDetails models in the repository
Add modelstore to the operation to allow backup/restore operations to update/query for corso models
Closes#268
* Factor out common code for getting kopia items
Both directory and single item restore in kopia need to do common tasks
like getting the item in question. Factor out that common code and
adjust tests to prep for directory restore.
Use a slice to back the data instead of adding directly to the channel
for two reasons (this may change in the future though):
* kopia loads all data about a directory at the same time
* consumers of the DataCollection may not pull items from the channel
at a fast rate, which could block adding to the channel. This could
lead to delays in discovering other directories to traverse in
multi-threaded scenarios
* Split KopiaWrapper into repo handle and logic
With ModelStore, multiple structs need a reference to the kopia repo.
Make a small wrapper class (conn) that can open and initialize a repo. The
wrapper handles concurrent closes and opens and does ref counting to
ensure it only drops the kopia handle when the last reference is closed.
Rename KopiaWrapper to Wrapper and keep backup/restore functionality
in it.
