## Description
This adds the following functionality to the CLI progress output:
- Add a message e.g. to describe a completed stage
- Add a message describing a stage in progress (using a spinner) and trigger completion
- Add a progress tracker (as a counter) for a specified number of items (when the # items is known up front
- Improves `ItemProgress` by aligning the columns a bit better and also removing the progress bar and replacing
with a byte counter
Finally - the above are used in the `backup` and `backup create onedrive` flows. Follow up PRs will wire these up for
exchange and the restore flows also.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1278
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a tag describing the service within the backup model and the backup details model. Adds a filter builder pattern to the store wrapper for filtering results according to certain tags. Finally, adds a filter builder to specify the service type when listing backups.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1226
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Extends progress bar display with multi-line support.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1112
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Adds a NoData status to operations in the event
that a backup (or, possibly in the future, restore)
finishes processing without having any items to
store or restore.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1000
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Adds some go runtime diagnostics tracking to
load testing, including some trace regioning.
Unfortunately, I couldn't find any third party trace library that didn't depend on a sidecar server
to sample against the application on. Therefore,
just starting with something basic.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Start-end event pairs want ids as a means to
match each pair. Restore operations now gen
a transient id for this purpose. Backup ops pre-
generate the base ID instead of waiting for the
model store to generate the ID at the end of
the operation.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #894
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Track the count of bytes read and written in
kopia. For backups, this means the count of
bytes fed into kopia (hashed bytes), and the
amount written after compression and dedupe
(total file bytes). For restore, this is the count of bytes in all files read.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #894
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Upgrades Graph API to the newest version. Breaking change from the previous version, all functionality needs to be inspected to ensure operation.
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #828<issue>
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Extend the kopia metrics to include the snapshot's TotalFileSize parameter. This value also gets used as the metric for BytesWritten at the end of a
backup.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #894
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Backup and Restore metrics need to know the
count of resource owners in the operation. This
processor extracts that data from the collections
that get passed from producer to consumer.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #894
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
1/ introduce an ID() method int the account
package which provides the tenantID of the
configured account provider.
2/ introduce a event bus mock for testing.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #741
## Test Plan
- [x] ⚡ Unit test
## Description
Adds events for repo init, backup start and end,
and restore start and end. Not all expected
values are available. TODOs have been marked
for values that need future implementation.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #741
## Test Plan
- [x] ⚡ Unit test
## Description
Wires up the OneDrive collection logic to `operation.Backup`
Includes an integration test that runs against the test domain
Two bug fixes:
- Skip the "root" item that is returned by the delta query
- Fix incorrect usage of the `filepath.SplitList` function which does
not split a path into components. Instead use `strings.Split`. This
is ok because the paths returned here are not OS specific.
Regardless - this logic will be refactored when we use the `path` pkg.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 🐹 Trivial/Minor
## Issue(s)
#548
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.
All other changes in the repository update the
package imports to the new module path.
* Fix wsl lint errors in pkg package
* Fix wsl lint errors in most of internal package
Leave some sub-packages out that have higher churn at the moment.
* Enable line width linter
Set to 120 which should be long enough to not be annoying but keep
things from getting "too long." Adding to get rid of the subjectiveness
of what is "too long." Tabs count as a single character.
* Turn on revive linter, ignore only a few things
* Fix lint errors
Ignore shadowing of 'suite' in tests for now. Also move some constants
that had the same value to tester.
In order for sdk users to build an operation using a repo,
without importing /inernal packages, the operation
options needs to be available via a /pkg package.
embedding a baseModel means that objects like backup
have a backup.StableID, but no backup.ID nor backup.
backupID. This could be confusing for users. This change
swaps the value and type names, so that baseModels
include an ID value of type StableID.
* store backup operation results in the backup manifest
Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry. These additional details will appear when users
call `corso backup list`.
* adds store package for wrapping model_store
Introduces the pkg/store package, which contains funcs
for wrapping the model_store with common requests.
This package choice was made for its combination
of being in an accessible place, centralizing functionality
and not introducing circular dependencies.
* add e2e backup-restore integration test
Adds an e2e integration test that starts by backing up
data, and ends with restoring it. Also makes various
amendments to other code where necessary to
facilitate this exercise.
* add output formatting control to cli
Adds the capacity for the CLI to output either a
text table or a json blob to the terminal. Table is
the default behavior, json is toggled with the --json
flag.
Update the backup operation to create RestorePoint and RestorePointDetails models in the repository
Add modelstore to the operation to allow backup/restore operations to update/query for corso models
Closes#268
* wire selectors up through backup handling
Selectors are implemented enough to add them end-
to-end in some places. This starts with backup
creation, since that's the most stable set of code in
the repo at the moment.
* Split KopiaWrapper into repo handle and logic
With ModelStore, multiple structs need a reference to the kopia repo.
Make a small wrapper class (conn) that can open and initialize a repo. The
wrapper handles concurrent closes and opens and does ref counting to
ensure it only drops the kopia handle when the last reference is closed.
Rename KopiaWrapper to Wrapper and keep backup/restore functionality
in it.
e2e wiring of persistence is not yet complete.
Will need modelstore integration, and additional
information about file and error counts from kw and gc.
* add operation results structs
Operations, both backup and restore, need to hold the
results of their operation, and be able to marshal the struct
to json for output.
* separate tenantID from m365 creds
Now that account.Account is in place, tenant id needs
to get removed from the credential set (it isn't actually
a secret) and placed in the account configuration instead.
* hook up restore end-to-end
Now that GC and KW both provide restore operations for a
single message, we can hook up the end-to-end restore
process. Integration tests for this change will follow in the
next PR.
Adds the RestoreOperation to the /internal/operations pkg.
restoreOp.Run() is a no-op at this time since data layer
packages are not prepared to hook up e2e.
* add backup to operations
Extend the operations set with a backupOperation. New backup
operations are created through an existing repository and, once
created, can be Run synchronously.
Integration tests have a skeleton, but the meat of them is on hold
while dependencies undergo implementation.
* remove operation progress
