* hook up restore end-to-end
Now that GC and KW both provide restore operations for a
single message, we can hook up the end-to-end restore
process. Integration tests for this change will follow in the
next PR.
Issue #214: Status of Connector
- Support package contains errors module
- status is updated after backup operation
- test suite added for new features.
* DataCollection and DataStream structs for kopia
* Helper function to get a single item of data
Returns a DataCollection with a single item.
* Test for backing up and restoring a single item
* add /pkg/source skeleton
Source acts as an intermediary between the
client and internal packages (GraphConnector, Kopia) to
specify the scope of data in a Backup or Restore
operation request.
* add Account pattern to mimic storage
Introduces /pkg/account, which produces a generic Account
struct that can be configured from specific provider
details (such as m365 credentials), and serialized back to
those details as needed.
Next steps include replacing repository.Account with the
new account. After that, tenantID should get removed from
the m365 credentials handling and placed into the Account
configuration instead.
Introduces a NewRestore method to the Repository struct.
cli/restore/exchange now generates a restoreOperation
and kicks off a Run() when `corso restore exchange` is called.
The restore operation still only accepts the placeholder
Target string slice.
Adds the RestoreOperation to the /internal/operations pkg.
restoreOp.Run() is a no-op at this time since data layer
packages are not prepared to hook up e2e.
Adds the `corso restore` command to the cli (which only displays
a help menu). And adds the `corso restore exchange`
command (which also only displays a help menu) to the cli.
Further functionality (like the complete flags) will come later.
* read test config from local file
Allows local configuration of the test environment by reading from
a .toml config file. If no file exists, the file read is a no-op. Value
prioritization is specified in the readTestConfig() func.
* use config file for backup create
Now that corso stores its prior connection details in a config
file, that file should get used to retrieve storage details as needed.
Fill in BackupCollection logic for kopia dirs and test
"Rehydrate" the given DataCollections into a hierarchy of kopia
virtualfs directories, allowing kopia to make a single snapshot of them.
Directory structure is extracted from the information returned by
DataCollection.FullPath().
Implements a mock exchange data collection that can be used to test without connecting to graph.
Currently returns a "fixed message" but we can extend this to return anything that implements io.Reader
e.g. a file that contains a serialized message.
DataCollection Feature added
In this merge, the DataCollection interface is defined for the GraphConnector. ExchangeDataCollection implements this interface. The subsequent definitions and test suites are included.
Adds unit tests for the config file read/write helpers.
It also uncovered a bug in how viper handles writing a config file when a config file path is set
directly. This required a workaround in our init logic when we are using a custom config file name.
This commit does the following:
- Refactors the init logic into a InitConfig helper
- Adds a unit test to validate basic ReadRepoConfig behavior
- Adds a unit test that uses WriteReadConfig to write config and ReadRepoConfig to read it
* Issue #144 Commit adds ability to add message id to errors
Commit related to Issue #140 to find cascading errors associated with
the failures GraphConnector queries.
* Issue 144: Update to go libraries to import multierror
* Reduced error package to include certain multierror interface.
* Issue #144: Added format interface for append. Added additional testing for multiple wrapping of error.
* Issue 133: Tests Failing.
Unable to troubleshoot the iteration error. Reflecting on an object that
is nil. Fundamentally, the code works in so far as the workflow. Initial
query finds the amount of emails, returns... This is where
DataCollection would be created, then the goroutine would have the
information it would need to run asynchronously.
* Issue #133: Populate DataCollection
Bug fix on NewPageIterator: Incorrect MailDiscriminatorValue.
DataCollection implemented. Brute force. However, this does not use the
populate method. The populate method to be added in the next step.
Test Suite: TestMailCount is an end-to-end test. Tests to see if the
amount of values in a mailbox are the amount of messages that are
retrieved. This is false at present. There are several errors from
ParseNodePackage: `unsupported AdditionalData type:
map[string]*jsonserialization.JsonParseNode`. Will have to determine how
errors are to be sent to the DataCollection. Issue #137. Upon
inspection, there are 59 messages on the test account. 7 of them receive
a parsing error and 2 are unaccounted for. Issue #138
* Issue #133: Collection passed into Serialize Message method.
* Issue #133: Changes for PR
Comments have been updated as requested. Additionally, input params have
been updated for clarity.
* Issue #133: Input structure for Options changed to string literal.
* Update src/internal/connector/graph_connector.go
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
* Update src/internal/connector/graph_connector.go
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
* Update src/internal/connector/graph_connector.go
Internal helper function populateCollection() removed as it was commented out anyway.
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
* Update src/internal/connector/graph_connector.go according to suggestion
Consolidate line 193 response.GetValue() into to for loop over folders
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
* Issue #133: Expected item removed
Expected Item can could change between the time that the DataCollection is created. Therefore, the channel will stay open until the object is filled. Definition updated.
* Issue #133: Updates made to graph_connector packages
- Test package updated to test private functions. Results in import
changes and calls to package level items. All tests verified as
operational prior to upload.
- GraphConnector removal of GetCount() for Mail objects.
ExchangeDataCollection call updated in response to definition change.
Small changes to call stack to remove troubleshooting stubs.
* Gofmt applied to exchange_data_collection.go
* Issue #133: Merge resolution
These changese resolves the issues with the merge that caused the testing to fail. Tests all passing and now merging back into trunk.
* PR Suggestion Get Length -> Length()
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
credentials requriements surface in many places thorughout corso:
they can be sourced from many locations (envs, files, manually),
and used in many more (cli, repo, kopia). This usage could blossom
into all kinds of duplicate structs sharing similar info. The goal
of this change is to centralize where credentials are declared
and managed, and how they then cascade out to other packages.
Add skeleton and test for backing up data with kopia
Full code for launching kopia snapshots is present but code for building
the directory structure to hand kopia is not. Therefore, no data will be
backed up right now.
* add backup to operations
Extend the operations set with a backupOperation. New backup
operations are created through an existing repository and, once
created, can be Run synchronously.
Integration tests have a skeleton, but the meat of them is on hold
while dependencies undergo implementation.
* remove operation progress
* adds env-based integration test controls
Some integration tests were running locally without being told
to do so via env variables. This adds some controllers for validating
if and when those tests should run, and for ensuring the env
variables necessary to run the test are available.
* testing helper comments, func name change
* add testing timeOf log, skip gc ci tests
* corrected the test skip in GC_test
* update kopia_test with helpers
* rename /testing files to *_test.go
This will ensure these files do not compile into the release binary.
* Revert "rename /testing files to *_test.go"
This reverts commit 04fd2046cc6d11bf3e3eb98d98a3e84b1bba6366.
POD struct that holds some simple information about a kopia upload. Not
wrapping or returning a handle to a kopia struct to keep upper layers of
corso clear of type dependencies. Struct will help with tests as it can
be returned from public functions in the KopiaWrapper package allowing
more black-box testing. Especially useful since we don't yet have a
restore flow, which means we can't test data == restore(backup(data)).
