Functions and interfaces moved to interact properly with connector package. Fields are no longer exported until the required functions are moved to the exchange package.
* store backup operation results in the backup manifest
Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry. These additional details will appear when users
call `corso backup list`.
* Basic golangci-lint config with gofmt and timeout
Remove corresponding sections from other CI config files.
* Enable comment and string spelling linter
* Fix spelling mistakes for linter
Co-authored-by: Danny <danny@alcion.ai>
* introduce exchange info selector support
Adds support in selectors/exchange for queries based on
backup.ExchangeInfo entries. This allows the declaration
of selectors based on non-identifier details such as sender,
subject, or receivedAt time.
Changes Exclude scope matching from being an Any-
match comparator to an All-match. This keeps exclude
and include behavior identical, hopefully making less
confusion for users.
* refactor selector scopes to accept slices
Cli flag implementation was showcasing a toil issue: building
selectors required a lot of repetitious code for combining
inputs into sets of scopes. Since all of these productions
were effectively identical (eg: for each user, then each folder,
create a scope with the ids), the cleaner solution is to pack
that behavior into the scope constructors themselves.
* add e2e backup-restore integration test
Adds an e2e integration test that starts by backing up
data, and ends with restoring it. Also makes various
amendments to other code where necessary to
facilitate this exercise.
Update the backup operation to create RestorePoint and RestorePointDetails models in the repository
Add modelstore to the operation to allow backup/restore operations to update/query for corso models
Closes#268
* wire selectors up through backup handling
Selectors are implemented enough to add them end-
to-end in some places. This starts with backup
creation, since that's the most stable set of code in
the repo at the moment.
A misuse of variable declaration that overlapped with
var shadowing on 'err' was causing the attachment retry
error to get lost, meaning failures to retrieve attachments
are occurring silently.
* Change DataCollection to return channel directly
Precursor to restoring multiple items from kopia. Allows one to keep a
DataCollection open until all items are processed without blocking
consumers of the DataCollection (they can use a select-block if needed).
* Update tests for new DataCollection interface
* Handle context cancellation with DataCollection
GraphConnector exports 2 error types. Recoverable and NonRecoverable. The package also implements error checks to confirm if errors are one of the exported types.
* separate tenantID from m365 creds
Now that account.Account is in place, tenant id needs
to get removed from the credential set (it isn't actually
a secret) and placed in the account configuration instead.
* Update how S3 storage structs are generated
* fix bug in printing year of date
* use the name of the test instead of trying to pull name from runtime
* always log the time when making the storage struct
* don't allow user to specify prefix
* Fixup tests for new test storage API
* Update function name and comment
* hook up restore end-to-end
Now that GC and KW both provide restore operations for a
single message, we can hook up the end-to-end restore
process. Integration tests for this change will follow in the
next PR.
Issue #214: Status of Connector
- Support package contains errors module
- status is updated after backup operation
- test suite added for new features.
* read test config from local file
Allows local configuration of the test environment by reading from
a .toml config file. If no file exists, the file read is a no-op. Value
prioritization is specified in the readTestConfig() func.
Implements a mock exchange data collection that can be used to test without connecting to graph.
Currently returns a "fixed message" but we can extend this to return anything that implements io.Reader
e.g. a file that contains a serialized message.
DataCollection Feature added
In this merge, the DataCollection interface is defined for the GraphConnector. ExchangeDataCollection implements this interface. The subsequent definitions and test suites are included.
* Issue #144 Commit adds ability to add message id to errors
Commit related to Issue #140 to find cascading errors associated with
the failures GraphConnector queries.
* Issue 144: Update to go libraries to import multierror
* Reduced error package to include certain multierror interface.
* Issue #144: Added format interface for append. Added additional testing for multiple wrapping of error.
* Issue 133: Tests Failing.
Unable to troubleshoot the iteration error. Reflecting on an object that
is nil. Fundamentally, the code works in so far as the workflow. Initial
query finds the amount of emails, returns... This is where
DataCollection would be created, then the goroutine would have the
information it would need to run asynchronously.
* Issue #133: Populate DataCollection
Bug fix on NewPageIterator: Incorrect MailDiscriminatorValue.
DataCollection implemented. Brute force. However, this does not use the
populate method. The populate method to be added in the next step.
Test Suite: TestMailCount is an end-to-end test. Tests to see if the
amount of values in a mailbox are the amount of messages that are
retrieved. This is false at present. There are several errors from
ParseNodePackage: `unsupported AdditionalData type:
map[string]*jsonserialization.JsonParseNode`. Will have to determine how
errors are to be sent to the DataCollection. Issue #137. Upon
inspection, there are 59 messages on the test account. 7 of them receive
a parsing error and 2 are unaccounted for. Issue #138
* Issue #133: Collection passed into Serialize Message method.
* Issue #133: Changes for PR
Comments have been updated as requested. Additionally, input params have
been updated for clarity.
* Issue #133: Input structure for Options changed to string literal.
* Update src/internal/connector/graph_connector.go
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
* Update src/internal/connector/graph_connector.go
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
* Update src/internal/connector/graph_connector.go
Internal helper function populateCollection() removed as it was commented out anyway.
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
* Update src/internal/connector/graph_connector.go according to suggestion
Consolidate line 193 response.GetValue() into to for loop over folders
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
* Issue #133: Expected item removed
Expected Item can could change between the time that the DataCollection is created. Therefore, the channel will stay open until the object is filled. Definition updated.
* Issue #133: Updates made to graph_connector packages
- Test package updated to test private functions. Results in import
changes and calls to package level items. All tests verified as
operational prior to upload.
- GraphConnector removal of GetCount() for Mail objects.
ExchangeDataCollection call updated in response to definition change.
Small changes to call stack to remove troubleshooting stubs.
* Gofmt applied to exchange_data_collection.go
* Issue #133: Merge resolution
These changese resolves the issues with the merge that caused the testing to fail. Tests all passing and now merging back into trunk.
* PR Suggestion Get Length -> Length()
Co-authored-by: Vaibhav Kamra <vkamra@alcion.ai>
