Use a slice to back the data instead of adding directly to the channel
for two reasons (this may change in the future though):
* kopia loads all data about a directory at the same time
* consumers of the DataCollection may not pull items from the channel
at a fast rate, which could block adding to the channel. This could
lead to delays in discovering other directories to traverse in
multi-threaded scenarios
A misuse of variable declaration that overlapped with
var shadowing on 'err' was causing the attachment retry
error to get lost, meaning failures to retrieve attachments
are occurring silently.
* Split KopiaWrapper into repo handle and logic
With ModelStore, multiple structs need a reference to the kopia repo.
Make a small wrapper class (conn) that can open and initialize a repo. The
wrapper handles concurrent closes and opens and does ref counting to
ensure it only drops the kopia handle when the last reference is closed.
Rename KopiaWrapper to Wrapper and keep backup/restore functionality
in it.
Tests that run multiple sub-tests do not use the fields in the test
suite because that would cause the model store instance to be reused
instead of having a new model store instance for each subtest.
* Implement ModelStore GetByType and Get
* Add tests for ModelStore Get functions
* Add stricter "type" checks for loaded models
Take modelType as parameter and check the model in question matches that
type. Adds a little extra layer of protection if models happen to have
the same struct layout.
e2e wiring of persistence is not yet complete.
Will need modelstore integration, and additional
information about file and error counts from kw and gc.
* Add ModelStore Update operation
* Tests for ModelStore Update function
* Add regression test for error during Update()
Ensure that if an error occurs during a ModelStore update operation the
previously stored model remains unchanged and no new model is visible to
the user.
* Simple Get and Put implementations
Get implementation is currently the one that uses the kopia ID of the
model/manifest.
* Basic tests for ModelStore Get/Put
* allow connect to create .corso config file
Current bug: if no .corso config file exists, then repo connect
will always fail, even if it has the correct details to build
a new config file. Solution: allow connect to build a .corso
config file when missing, so long as the operation succeeds
otherwise.
In tandem, return an error whenever a user attempts to
call repo connect with details that do not match the existing
.corso config file.
* add operation results structs
Operations, both backup and restore, need to hold the
results of their operation, and be able to marshal the struct
to json for output.
* Change DataCollection to return channel directly
Precursor to restoring multiple items from kopia. Allows one to keep a
DataCollection open until all items are processed without blocking
consumers of the DataCollection (they can use a select-block if needed).
* Update tests for new DataCollection interface
* Handle context cancellation with DataCollection
GraphConnector exports 2 error types. Recoverable and NonRecoverable. The package also implements error checks to confirm if errors are one of the exported types.
* separate tenantID from m365 creds
Now that account.Account is in place, tenant id needs
to get removed from the credential set (it isn't actually
a secret) and placed in the account configuration instead.
* Update how S3 storage structs are generated
* fix bug in printing year of date
* use the name of the test instead of trying to pull name from runtime
* always log the time when making the storage struct
* don't allow user to specify prefix
* Fixup tests for new test storage API
* Update function name and comment
* move config unions to common code
The configuration union handlers in Storage and Account
overlapped significantly in behavior. Moving those helpers into
a common code folder was requested. Although the behavior
was similar across the files, the types were not, requiring
the addition of generics to solve the need.
* hook up restore end-to-end
Now that GC and KW both provide restore operations for a
single message, we can hook up the end-to-end restore
process. Integration tests for this change will follow in the
next PR.
Issue #214: Status of Connector
- Support package contains errors module
- status is updated after backup operation
- test suite added for new features.
* DataCollection and DataStream structs for kopia
* Helper function to get a single item of data
Returns a DataCollection with a single item.
* Test for backing up and restoring a single item
Adds the RestoreOperation to the /internal/operations pkg.
restoreOp.Run() is a no-op at this time since data layer
packages are not prepared to hook up e2e.
Adds the `corso restore` command to the cli (which only displays
a help menu). And adds the `corso restore exchange`
command (which also only displays a help menu) to the cli.
Further functionality (like the complete flags) will come later.
* read test config from local file
Allows local configuration of the test environment by reading from
a .toml config file. If no file exists, the file read is a no-op. Value
prioritization is specified in the readTestConfig() func.
* use config file for backup create
Now that corso stores its prior connection details in a config
file, that file should get used to retrieve storage details as needed.
Fill in BackupCollection logic for kopia dirs and test
"Rehydrate" the given DataCollections into a hierarchy of kopia
virtualfs directories, allowing kopia to make a single snapshot of them.
Directory structure is extracted from the information returned by
DataCollection.FullPath().
Implements a mock exchange data collection that can be used to test without connecting to graph.
Currently returns a "fixed message" but we can extend this to return anything that implements io.Reader
e.g. a file that contains a serialized message.
DataCollection Feature added
In this merge, the DataCollection interface is defined for the GraphConnector. ExchangeDataCollection implements this interface. The subsequent definitions and test suites are included.
* Issue #144 Commit adds ability to add message id to errors
Commit related to Issue #140 to find cascading errors associated with
the failures GraphConnector queries.
* Issue 144: Update to go libraries to import multierror
* Reduced error package to include certain multierror interface.
* Issue #144: Added format interface for append. Added additional testing for multiple wrapping of error.
