## Description
Replace all cases of context.Background() and
context.TODO() with the testing ctx, which seeds
a logging instance into the context. The seed is
less important than the flush action, since errant log.Ctx(ctx) calls will automatically use the log
singleton. But in order for the logs to show up
within the test, they need to get flushed.
See my comments for focus on non-chaff changes.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Adds a NoData status to operations in the event
that a backup (or, possibly in the future, restore)
finishes processing without having any items to
store or restore.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1000
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Exchange app e2e tests need to scope their
backup dataset to only the primary folder, else
we risk exponential duplication of data when
the next backup picks up all the corso-copied
folders.
## Type of change
- [x] 🌻 Feature
- [x] 🐛 Bugfix
## Issue(s)
* #501
### Awaiting completion
- #810
- #811
## Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
Start-end event pairs want ids as a means to
match each pair. Restore operations now gen
a transient id for this purpose. Backup ops pre-
generate the base ID instead of waiting for the
model store to generate the ID at the end of
the operation.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #894
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Track the count of bytes read and written in
kopia. For backups, this means the count of
bytes fed into kopia (hashed bytes), and the
amount written after compression and dedupe
(total file bytes). For restore, this is the count of bytes in all files read.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #894
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Extend the kopia metrics to include the snapshot's TotalFileSize parameter. This value also gets used as the metric for BytesWritten at the end of a
backup.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #894
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Backup and Restore metrics need to know the
count of resource owners in the operation. This
processor extracts that data from the collections
that get passed from producer to consumer.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #894
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
1/ introduce an ID() method int the account
package which provides the tenantID of the
configured account provider.
2/ introduce a event bus mock for testing.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #741
## Test Plan
- [x] ⚡ Unit test
## Description
Adds events for repo init, backup start and end,
and restore start and end. Not all expected
values are available. TODOs have been marked
for values that need future implementation.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #741
## Test Plan
- [x] ⚡ Unit test
## Description
Wires up the OneDrive collection logic to `operation.Backup`
Includes an integration test that runs against the test domain
Two bug fixes:
- Skip the "root" item that is returned by the delta query
- Fix incorrect usage of the `filepath.SplitList` function which does
not split a path into components. Instead use `strings.Split`. This
is ok because the paths returned here are not OS specific.
Regardless - this logic will be refactored when we use the `path` pkg.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
#548
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.
All other changes in the repository update the
package imports to the new module path.
## Description
Exchange events were amended to use calendars
as a folder structure. This updates the selector
to treat events as having folders similar to mail and
contacts.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
* Fix wsl lint errors in pkg package
* Fix wsl lint errors in most of internal package
Leave some sub-packages out that have higher churn at the moment.
In order to both maintain control of test env data, as
well as ensure open-source users are able to run the
integration tests on personal account data, the test
user must be derived from the test configuration, rather
than hardcoded within the tests.
The alias of testing to ctesting prevented auto-importing
of the testing library. This change (arbitrarily) renames
the package so that it doesn't collide with the core pkg
for "testing".
In order for sdk users to build an operation using a repo,
without importing /inernal packages, the operation
options needs to be available via a /pkg package.
* store backup operation results in the backup manifest
Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry. These additional details will appear when users
call `corso backup list`.
* refactor selector scopes to accept slices
Cli flag implementation was showcasing a toil issue: building
selectors required a lot of repetitious code for combining
inputs into sets of scopes. Since all of these productions
were effectively identical (eg: for each user, then each folder,
create a scope with the ids), the cleaner solution is to pack
that behavior into the scope constructors themselves.
* adds store package for wrapping model_store
Introduces the pkg/store package, which contains funcs
for wrapping the model_store with common requests.
This package choice was made for its combination
of being in an accessible place, centralizing functionality
and not introducing circular dependencies.
* add e2e backup-restore integration test
Adds an e2e integration test that starts by backing up
data, and ends with restoring it. Also makes various
amendments to other code where necessary to
facilitate this exercise.
Update the backup operation to create RestorePoint and RestorePointDetails models in the repository
Add modelstore to the operation to allow backup/restore operations to update/query for corso models
Closes#268
* wire selectors up through backup handling
Selectors are implemented enough to add them end-
to-end in some places. This starts with backup
creation, since that's the most stable set of code in
the repo at the moment.
* Split KopiaWrapper into repo handle and logic
With ModelStore, multiple structs need a reference to the kopia repo.
Make a small wrapper class (conn) that can open and initialize a repo. The
wrapper handles concurrent closes and opens and does ref counting to
ensure it only drops the kopia handle when the last reference is closed.
Rename KopiaWrapper to Wrapper and keep backup/restore functionality
in it.
e2e wiring of persistence is not yet complete.
Will need modelstore integration, and additional
information about file and error counts from kw and gc.
* add operation results structs
Operations, both backup and restore, need to hold the
results of their operation, and be able to marshal the struct
to json for output.
* separate tenantID from m365 creds
Now that account.Account is in place, tenant id needs
to get removed from the credential set (it isn't actually
a secret) and placed in the account configuration instead.
* Update how S3 storage structs are generated
* fix bug in printing year of date
* use the name of the test instead of trying to pull name from runtime
* always log the time when making the storage struct
* don't allow user to specify prefix
* Fixup tests for new test storage API
* Update function name and comment
* read test config from local file
Allows local configuration of the test environment by reading from
a .toml config file. If no file exists, the file read is a no-op. Value
prioritization is specified in the readTestConfig() func.
* add backup to operations
Extend the operations set with a backupOperation. New backup
operations are created through an existing repository and, once
created, can be Run synchronously.
Integration tests have a skeleton, but the meat of them is on hold
while dependencies undergo implementation.
* remove operation progress
