116 Commits

Author SHA1 Message Date
Danny
bd559f690e
Exchange service functions (#454)
`exchange`.service_actions.go created. Create/Delete Mail folder functions inherited. Test suite updated accordingly.
2022-08-01 16:29:03 +00:00
Danny
2f56a849c9
Exchange.Collection.Items() to initiate populate method (#446)
`Items()` starts populate method for collection for the `exchange.Collection` package. Test packages expanded to verify the functionality of this feature.
2022-07-31 17:57:14 +00:00
Keepers
7c62939c85
update GC's restore to use AwaitStatus (#451)
GC's restore funcs update their status field differently than
the backup funcs.  This brings the behavior in line.
2022-07-29 21:09:15 +00:00
Danny
da9a22271d
Mock Connector expansion (#437)
Mock Connector data collection is a Messageable item and has been changed to be utilized in other portions of the corso repo.
2022-07-29 15:08:36 -04:00
Danny
5a68108006
Gc separate tests (#448)
Merge places online and offline tests into different files.
2022-07-29 11:33:15 -06:00
Danny
e6f5b4515d
GC: exchange.Collection PopulateFunction refactor without TaskList (#441)
exchange.PopulateFunc executed without a tasklist.
2022-07-29 12:35:27 -04:00
Danny
9eb6a5dd0c
GC: Tasklist removed from packages (#440)
Tasklist struct removed from code base
2022-07-29 12:32:58 +00:00
Danny
992504ed36
exchange.Collection FinishPopulation function removed (#436)
* removal of FinishPopulation() method.
* graph_connector disconnected from exchange collection internal functions.
2022-07-28 14:08:35 -04:00
Danny
a4d99a74ed
Adapter creation moved to graph package (#432)
CreateAdapter function exported to the `graph` package. This will be the same area that authentication and token refreshment will eventually go.
2022-07-28 16:06:33 +00:00
Danny
987980510c
exchange.Collection addition of job field to struct (#434)
`exchange.Collection` field **jobs** `type: [] string` added to struct
2022-07-28 15:50:55 +00:00
Danny
402b3aae06
Exported fields removed (#433)
Exported fields removed.
2022-07-28 15:22:28 +00:00
Danny
8bfff3c88f
Population Function Moved to Exchange Package (#429)
Functions and interfaces moved to interact properly with connector package. Fields are no longer exported until the required functions are moved to the exchange package.
2022-07-27 19:18:20 -04:00
Danny
665fa21b13
ExchangeDataCollection moved to exchange.ObjectCollection (#423)
Exchange DataCollection moved to the exchange package. Unused fields removed at this time.
2022-07-27 21:22:27 +00:00
Danny
34a7a1a80c
Data Collection --> Collection refactor (#415)
DataCollection changed to Collection in the repository. All associated imports changed to reflect the change.
2022-07-27 12:04:31 -04:00
Danny
3e792e69eb
Move DataCollection to data package (#414)
DataCollection moved to `/src/internal/data` directory and imports changed to recognize the move.
2022-07-27 08:04:35 -04:00
Keepers
587c239dd1
store backup operation results in the backup manifest (#410)
* store backup operation results in the backup manifest

Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry.  These additional details will appear when users
call `corso backup list`.
2022-07-26 19:35:14 +00:00
Keepers
5a532e808e
fix errant conditional on GC restore success counter (#407) 2022-07-25 13:14:13 -06:00
Danny
25c781eec6
Gc edc extension (#402)
* Extension of ExchangeDataCollection for async population.
2022-07-25 12:46:17 -04:00
Keepers
dcbc149aa7
Build binary test separately (#399)
* refactor ci caching to ensure go test runs without downloading packages
2022-07-25 09:05:40 -06:00
ashmrtn
395c7c8525
Add spell check linter and run gofmt in golangci-lint (#396)
* Basic golangci-lint config with gofmt and timeout

Remove corresponding sections from other CI config files.

* Enable comment and string spelling linter

* Fix spelling mistakes for linter

Co-authored-by: Danny <danny@alcion.ai>
2022-07-22 18:28:25 +00:00
Keepers
532922f662
introduce exchange info selector support (#379)
* introduce exchange info selector support

Adds support in selectors/exchange for queries based on
backup.ExchangeInfo entries.  This allows the declaration
of selectors based on non-identifier details such as sender,
subject, or receivedAt time.

Changes Exclude scope matching from being an Any-
match comparator to an All-match.  This keeps exclude
and include behavior identical, hopefully making less
confusion for users.
2022-07-21 15:08:03 -06:00
Danny
dfcba3bf0a
Fail-fast option added (#377)
Fail-fast option added.
2022-07-21 14:59:11 +00:00
Danny
fe92d7c9db
Issue #359: create folder enhancement added. (#375)
Issue #359: create folder feature added.
Feature required to finish task #374
2022-07-21 13:32:53 +00:00
Vaibhav Kamra
1a31a07425
Populate ExchangeInfo correctly (#373)
Message Info added back to package.
2022-07-20 08:30:28 -04:00
Keepers
746c88a233
refactor selector scopes to accept slices (#352)
* refactor selector scopes to accept slices

Cli flag implementation was showcasing a toil issue: building
selectors required a lot of repetitious code for combining
inputs into sets of scopes.  Since all of these productions
were effectively identical (eg: for each user, then each folder,
create a scope with the ids), the cleaner solution is to pack
that behavior into the scope constructors themselves.
2022-07-19 16:04:07 -06:00
Keepers
6d04c82992
correct count of successful status items (#366)
* correct count of successful status items
2022-07-19 13:53:42 -06:00
Danny
d00332f328
GC async population of m365 objects mail (#332)
-- Changes made to incorporate async population of a DataCollection
2022-07-18 21:01:58 +00:00
Danny
5a9f2e4601
GC custom errors to inherit from internal/common (#337)
Custom error package added and extended to the internal/connector package
2022-07-15 21:21:11 +00:00
Keepers
105fd7383a
add e2e backup-restore integration test (#311)
* add e2e backup-restore integration test

Adds an e2e integration test that starts by backing up
data, and ends with restoring it.  Also makes various
amendments to other code where necessary to
facilitate this exercise.
2022-07-14 09:00:55 -06:00
Danny
b9652c66e3
GC Serialization fix: event message support (#304)
Support for eventMessage Request and Response capabilities added for serialization.
2022-07-11 13:04:20 -04:00
Vaibhav Kamra
fa190da682
Rename restorepoint to backup (#314) 2022-07-11 16:06:04 +00:00
Danny
60ab132960
GC: Create Consistent Snapshot (#292)
GraphConnector: Query consistency feature added
See document for implementation details: [document](https://www.notion.so/alcion/GraphConnector-RestorePoint-Consistency-for-Queries-81cce329c6e84cd79121cda0d511e1f6)
2022-07-08 15:45:25 -04:00
Vaibhav Kamra
6e9bd634e4
Create RestorePoint and RestorePointDetails (#290)
Update the backup operation to create RestorePoint and RestorePointDetails models in the repository

Add modelstore to the operation to allow backup/restore operations to update/query for corso models

Closes #268
2022-07-08 01:41:40 +00:00
Vaibhav Kamra
9d21d65bc7
Use MockExchangeData in kopia backup tests (#303)
This implements the DataStreamInfo interface needed by BackupCollections
2022-07-08 00:23:34 +00:00
Keepers
c29a4fffe0
wire selectors up through backup handling (#278)
* wire selectors up through backup handling

Selectors are implemented enough to add them end-
to-end in some places.  This starts with backup
creation, since that's the most stable set of code in
the repo at the moment.
2022-07-07 17:02:25 -06:00
Keepers
503fca14f5
use gc status to populate op results (#287)
Exports access to the graphconnector status prop
so that restore and backup ops can track those metrics.
2022-07-07 13:10:45 -06:00
Keepers
e8ec528113
break gc iterator cb to own func, fix err scope (#282)
A misuse of variable declaration that overlapped with
var shadowing on 'err' was causing the attachment retry
error to get lost, meaning failures to retrieve attachments
are occurring silently.
2022-07-06 12:49:32 -06:00
ashmrtn
1c6f0994e4
Switch DataCollection to return a channel (#244)
* Change DataCollection to return channel directly

Precursor to restoring multiple items from kopia. Allows one to keep a
DataCollection open until all items are processed without blocking
consumers of the DataCollection (they can use a select-block if needed).

* Update tests for new DataCollection interface

* Handle context cancellation with DataCollection
2022-06-28 12:48:24 -07:00
Danny
3ee7ff0c0b
Recoverable and Non-Recoverable errors (#232)
GraphConnector exports 2 error types. Recoverable and NonRecoverable. The package also implements error checks to confirm if errors are one of the exported types.
2022-06-27 20:29:15 -04:00
Keepers
689c5cc1e9
separate tenantID from m365 creds (#231)
* separate tenantID from m365 creds

Now that account.Account is in place, tenant id needs
to get removed from the credential set (it isn't actually
a secret) and placed in the account configuration instead.
2022-06-27 13:54:37 -06:00
ashmrtn
902294b70d
S3 test prefix naming (#228)
* Update how S3 storage structs are generated

  * fix bug in printing year of date
  * use the name of the test instead of trying to pull name from runtime
  * always log the time when making the storage struct
  * don't allow user to specify prefix

* Fixup tests for new test storage API

* Update function name and comment
2022-06-23 11:55:26 -07:00
Keepers
2b65ff80f2
hook up restore end-to-end (#226)
* hook up restore end-to-end

Now that GC and KW both provide restore operations for a
single message, we can hook up the end-to-end restore
process.  Integration tests for this change will follow in the
next PR.
2022-06-22 18:17:26 -06:00
Danny
128e9274e9
GraphConnector Status support module (#222)
Issue #214: Status of Connector
- Support package contains errors module
- status is updated after backup operation
- test suite added for new features.
2022-06-22 17:17:36 -04:00
Keepers
243343c0e9
Replace fmt prints with the logger (#220)
Now that we have the logger in place, we should use it
in place of fmt.Print calls for development and debugging output.
2022-06-17 13:45:22 -06:00
Danny
81bdc69c5a
Attach retry (#202)
Merge of Attach retry branch adds inline retry for attachments. Appropriate tests included.
2022-06-16 07:05:29 -04:00
ashmrtn
bf267a5478
Return io.ReadCloser from interface (#209)
Kopia returns io.ReadCloser which implies that some objects may need to
reclaim resources. Expose that functionality here.
2022-06-15 14:38:28 -07:00
ashmrtn
4f2ab23fbd
Add "mail" as part of FullPath for dir structure (#201)
* Add "mail" as part of FullPath for dir structure

* Update restore code for new dir structure
2022-06-15 11:59:38 -07:00
Danny
e0199b6b26
Issue 184 Full Path added to DataCollection (#199)
Merge of Branch adds FullPath to the DataCollection() where each DataCollection is based on the folder. Thus, returning a [] DataCollection
2022-06-14 17:09:02 -04:00
Danny
88bc374208
Issue 181: GC Implement restore (#197)
Addition of restoreMessages(dc DataCollection) error for GraphConnector
Merge completes Issue #181. Test suites updated
2022-06-14 14:57:38 -04:00
Keepers
ffdb469cf6
Issue 193 (#194)
* add integration tests to backupop run
2022-06-13 15:45:07 -06:00