327 Commits

Author SHA1 Message Date
Keepers
105fd7383a
add e2e backup-restore integration test (#311)
* add e2e backup-restore integration test

Adds an e2e integration test that starts by backing up
data, and ends with restoring it.  Also makes various
amendments to other code where necessary to
facilitate this exercise.
2022-07-14 09:00:55 -06:00
Keepers
39c85e1a84
add output formatting control to cli (#329)
* add output formatting control to cli

Adds the capacity for the CLI to output either a
text table or a json blob to the terminal.  Table is
the default behavior, json is toggled with the --json
flag.
2022-07-13 14:15:08 -06:00
Vaibhav Kamra
e5f5c38d4e
Add tools dependency (#313)
Adding tools.go allows us to declare a dependency on the stringer package (used for go generate)

Reference: https://play-with-go.dev/tools-as-dependencies_go115_en/
2022-07-11 08:47:15 -07:00
Danny
60ab132960
GC: Create Consistent Snapshot (#292)
GraphConnector: Query consistency feature added
See document for implementation details: [document](https://www.notion.so/alcion/GraphConnector-RestorePoint-Consistency-for-Queries-81cce329c6e84cd79121cda0d511e1f6)
2022-07-08 15:45:25 -04:00
Vaibhav Kamra
0a5aa8ce73
Wire up RestorePoints to backup list (#305)
Currently displays the restorepoints in tablular form. We can add json output in a follow-up using the segment-cli library.

Closes #269
2022-07-08 09:12:43 -06:00
ashmrtn
6bee4e5b41
Use latest kopia version in corso (#274)
newest release includes all changes we currently require
2022-07-05 10:57:23 -07:00
Keepers
ce52cc5f41
Add the backup list command to the cli (#254)
* Add the backup list command to the cli

Adds `corso backup list <service>` to the cli commands.
Currently does nothing, while we're waiting on
downstream implementation.  e2e hookup will arrive later.
2022-06-30 09:30:37 -06:00
Keepers
689c5cc1e9
separate tenantID from m365 creds (#231)
* separate tenantID from m365 creds

Now that account.Account is in place, tenant id needs
to get removed from the credential set (it isn't actually
a secret) and placed in the account configuration instead.
2022-06-27 13:54:37 -06:00
Keepers
108065df75
remove repository.Account (#229)
* remove repository.Account

Now that account.Account exists, we can replace the
deprecated repository.Account struct with the account
version.
2022-06-23 16:02:12 -06:00
Keepers
fcdb42bc89
add restore and restore exchange cli (#210)
Adds the `corso restore` command to the cli (which only displays
a help menu).  And adds the `corso restore exchange`
command (which also only displays a help menu) to the cli.
Further functionality (like the complete flags) will come later.
2022-06-15 15:46:09 -06:00
ashmrtn
0e4e9e8fd6
Use upstream kopia as our import (#205)
Still need to use a pseudo-version because no tagged version of kopia
has the features we need.
2022-06-15 12:00:43 -07:00
Keepers
f03432e303
read test config from local file (#179)
* read test config from local file

Allows local configuration of the test environment by reading from
a .toml config file.  If no file exists, the file read is a no-op.  Value
prioritization is specified in the readTestConfig() func.
2022-06-10 13:21:03 -06:00
Vaibhav Kamra
0da0c786b5
Mock ExchangeDataCollection (#166)
Implements a mock exchange data collection that can be used to test without connecting to graph.

Currently returns a "fixed message" but we can extend this to return anything that implements io.Reader
e.g. a file that contains a serialized message.
2022-06-08 16:33:59 -07:00
Keepers
2104e66fea
add graph connector to backupOp (#165)
Adds graph connector to the backup operation implementation.
This includes feeding the account credentials into the
backup as well.
2022-06-08 17:32:14 -06:00
ashmrtn
33e8e978ba
Use version of kopia with StreamingDirectory (#156)
* Use pseudo-version to pull fork of kopia

Directs to local fork of kopia that has support for
StreamingDirectories.
2022-06-08 09:49:10 -07:00
Vaibhav Kamra
cbbd5ed0bb
Fix AWS access key override logic (#159) 2022-06-08 08:51:32 -06:00
Danny
ddc9cd2c25
Issue #144 Commit adds ability to add message id to errors (#151)
* Issue #144 Commit adds ability to add message id to errors
Commit related to Issue #140 to find cascading errors associated with
the failures GraphConnector queries.

* Issue 144: Update to go libraries to import multierror

* Reduced error package to include certain multierror interface.

* Issue #144: Added format interface for append. Added additional testing for multiple wrapping of error.
2022-06-08 08:59:59 -04:00
Vaibhav Kamra
dbb778c569
Initialize a config file for the CLI (#148)
Initialize a config file
2022-06-07 08:53:58 -06:00
Keepers
6e71e44024
introduce operations (#114) (#122)
* introduce operations (#114)

Adds the /internal/operations package.  This is a starting point
which only includes the base operation struct and handling.
Per-process (backup/restore) operations to come at a later time.

* adding comments for clarity in op_progress
2022-06-03 15:50:13 -06:00
Vaibhav Kamra
c33ef8631b
ExchangeDataCollection placeholder (#123)
Introduces ExchangeDataCollection and a method on GraphConnector to instantiate one for a specified user

Refer to https://www.notion.so/alcion/GraphConnector-interfaces-eadd11a49542436cb611da0135dda7cb
2022-06-03 13:43:19 -07:00
Danny
01c275707a
GraphConnector initialization (#104)
* Issue #68: Updates to dependency files.

* Issue #68: Added GraphConnector module and test suite.

* Issue #68: ErrorList module and test added to package to support GraphConnector.

* Issue #68: Updates made to address comments.

* Issue #68: Added additonal comments to structs and corrected library changes.

* Issue #68: Added additonal comments to structs and corrected library changes.

* Issue #68: Updates made to package.

* Update errorList.go

Fix typo

* Issue #68: ErrorList struct removal required
The structure of the errorList is not required for the first
initialization of the GraphConnector. In the case that this structure or
addtional functionality needs to be added in the future, an issue will
be created.

* Issue #68: Simplification and structure change
The errorList has been changed to []errors. Additionally, the
credentials and authentication have been changed to inline. Provided
early exits for creating NewGraphConnectors. Note, adapter objects can
be created with strings that fail authentication. Reduced the amount of logs.

* Issue #68: Prepend
[errors](https://github.com/alcionai/corso/pull/104#discussion_r887268576)
This fixes the bug that identified in the PR.

* Update src/internal/connector/graphConnector.go

Adjustment for styling.

Co-authored-by: Keepers <104464746+ryanfkeepers@users.noreply.github.com>

* Update src/internal/connector/graphConnector_test.go

Issue #68: PR Resolution
Test environmental value changed to be consistent with the rest of the package.

Co-authored-by: Keepers <104464746+ryanfkeepers@users.noreply.github.com>

* Issue #68: Resolving Open Questions from reviewers
Prior to commit, Issue #116. Additionally, the entire error package was
requested to be removed from the component. GraphConnector tests are
preliminary set-up to investigate the Issue #116. The code will be
redacted after the new functionality is verified. Stashing for merge for
the changes from the reviewers.

* Issue #116: Bug fix. Response verified not to be nil prior to using it in next step.

* Issue #116: Added table structure to verify GraphConnector can handle being given an empty set of parameters.

* Issue #68: PR Changes
List of changes:
- Added check to ensure conversion of user object passed
- setTenant not exported
- TODO added user query during initialization
- Iteration flow adjusted for paging.

Co-authored-by: Keepers <104464746+ryanfkeepers@users.noreply.github.com>
2022-06-03 02:46:15 -04:00
Keepers
535cb9e1f5
add common config and encryption passwd (#74) (#99)
* add common config and encryption passwd

Adds a provider-independent configuration handler, and the
the encryption password config property.  The password is used
to encrypt and decrypt the kopia repository properties file.

* fix corso_password in ci.yml

* actually use the corso password in testing

* replace passwd in ci.yml with a secret

* ci.yml secret typo fix
2022-06-01 16:00:41 -06:00
Vaibhav Kamra
d34175bd19
Use testify for repository tests (#88) 2022-05-27 10:25:24 -07:00
Keepers
569ba524ac
adds kopia pkg to handle integration (#25) (#76)
* adds kopia pkg to handle integration (#25)

internal/kopia will be used to abstract all kopia integration in a
central location.

* defer blob.Storage closure in kopia

blob.Storage objects must get closed at the end of their usage.
This currently isn't getting called.
2022-05-25 13:08:47 -06:00
Keepers
2ae4f4cdcb
introduce repository api scaffold with s3 (#6) (#54)
* introduce repository api scaffold with s3 (#6)

introduces the public repository package in /pkg, with a
skeleton implementation for generating the s3 provider config
and boilerplate for where that usage gets centralized.

* condense init/connect to single action

A double-pass NewRepo() && Connect() allows Repository structs
to exist independent guarantees that their usage will be successful.
This hides the struct generation within the Connect and Initialize
funcs so that the construction and guarantees can occur as
a single process.
2022-05-19 16:05:33 -06:00
Keepers
a352ddf8a1
add coba and cli to corso (#30)
Adds the spf13/cobra cli handling package.  Main() now
processes cli commands defined by the /cli package.
2022-05-09 13:36:44 -06:00
Ashlie Martinez
0734a990d2 Move/update gomode file.
Now reflects the fact that this will be cloned from github. Also place
in source directory so that all packages in the project don't need to be
`src/foo`.
2022-05-03 16:12:30 -07:00