Rename backup model variables that holds the protected resource ID and
name. The variable name has changed from ResourceOwner[ID|Name] to
ProtectedResource[ID|Name].
The previous version of the variable in the model of the form
ProtectedResourceX wasn't getting populated and never had any data
persisted for it even though other code sometimes relied on it having a
value.
The current update ** DOES NOT ** change the JSON tag associated with
the renamed variable. This means that golang's stdlib JSON serialization
with continue to use the old name. Thus there's no need for additional
code to handle previously serialized backup models.
This was an accidental mis-use of the skipped item pattern. Items deleted during in flight during backup due to race conditions do not count as permanent
skips in the way that other skipped items do.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#4044
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
Currently the backup output show,
- total backup size - size of files + size of metafiles
- no of files - no of files + no of metafile
With this change the output of backup command shows size of only files and count include only file count.
NOTE: all current three services results will be impacted here.
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3304
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3478
#### Test Plan
- [x] ⚡ Unit test
moves common/time.go to common/dttm/dttm.go, and renames many of the consts and funcs with names that better explain their design and intent of usage.
Only code movement, no logical changes.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Prints backup stats to the CLI following completion. In case of multiple users, the stats for each backup is printed at the end of the backup, rather than at the end of the command.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🌻 Feature
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
Adds two maps to resource-owner handling:
id-to-name and name-to-id. Expectation is that
these maps will either get populated by a caller
as a pre-process before initializing the gc client, or
gc will (later pr) be able to look up the owner and
populate those maps itself. The maps are
used to set the selector id and name for iface
compliance. Only supported by exchange in this PR.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2825
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
OneNote files >2GB cannot be downloaded via the graph API. This change will gracefully skip them and prevent the backup from erroring out.
A OneNote file is represented by a Package(folder) which contains two file within it, `<file>.one` and `<file>.onetoc2`. From what I can tell the `onetoc2` file is the metadata file and the `one` file is the one that contains data and can get big.
**The current implementation has a limitation(or feature depending on how you see it) in that we will end up doing a partial backup with just the `onetoc2` file but avoiding the big `one` file if that is big.**
I did not try to change the behaviour as even currently if for somehow we ended up with an error like 404 when trying to backup one of these files, we do skip it. I've added a comment in for now, let me know if we should go back and revisit the behaviour. *The main issue is the fact that we only hit this problem once it reaches kopia and removing it at that point is harder.*
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2910
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Treat a 404 as an unavailable item and skip it similar to malware items.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Treat a 404 as an unavailable item and skip it similar to malware items.
Adds the item struct to the fault package for tracking serializable and dedupliatable error sources.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
#### Issue(s)
* #2708
#### Test Plan
- [x] ⚡ Unit test
## Description
Now that fault errors is in place, we don't need
the operation stats errs.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Leave out account and storage subpackages as those are used by tester and create an import cycle.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2373
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Renaming the funcs in the fault
package to be more clear about
their purpose and behavior. Largely
just find&replace changes, except
for fault.go and the fault examples.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
This adds information about owner of the backup to `--json` output of backup list. Previously we were showing owner in the normal backup list command, but not when asked to as json.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds the fault.Errors struct (now exported) to the
operations base. stats.Errs is retained in the
backup and restore wrappers to avoid breaking
changes and allow for deserialization. We will
continue to use the current error return until
dependencies are fully updated to use Errors.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
The selectors printable code wasn't being used in any valid way, so that's out. Backup List printing now refers to the DiscreteOwner in the embedded selector as the ResourceOwner reference. Renamed the "Selectors"
column in the List print table to "Resource Owner" to better match what should be contained in that field.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
## Type of change
- [x] 🐛 Bugfix
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1617
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Now that resource owners are identified via
the selector itself, rather than each scope, we
can remove the resource owner data from
scope production and data.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
selector creation now includes a parameter for
a slice of resource owners (users or sites). This
is step one in migrating resource owner lists out
of scopes and into the selector. next step is to
have the selector utilize the primary list instead
of the per-scope list.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Add size and stored size to cli output for `backup list`
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/1622
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds a tag describing the service within the backup model and the backup details model. Adds a filter builder pattern to the store wrapper for filtering results according to certain tags. Finally, adds a filter builder to specify the service type when listing backups.
## Type of change
- [x] 🐛 Bugfix
## Issue(s)
* #1226
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
## Description
Track the count of bytes read and written in
kopia. For backups, this means the count of
bytes fed into kopia (hashed bytes), and the
amount written after compression and dedupe
(total file bytes). For restore, this is the count of bytes in all files read.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #894
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.
All other changes in the repository update the
package imports to the new module path.
* Fix wsl lint errors in pkg package
* Fix wsl lint errors in most of internal package
Leave some sub-packages out that have higher churn at the moment.
embedding a baseModel means that objects like backup
have a backup.StableID, but no backup.ID nor backup.
backupID. This could be confusing for users. This change
swaps the value and type names, so that baseModels
include an ID value of type StableID.
* store backup operation results in the backup manifest
Adds backup operation metadata like the outcome
statistics and selector definitions to the backup manifest
entry. These additional details will appear when users
call `corso backup list`.
* filter backup details by flags
`backup details` should have its output filtered by the flags provided by
the user. In addition, the selector's FilterDetails should maintain
information (esp service info) about the entries, rather than slicing them
down to only the path reference.
* add output formatting control to cli
Adds the capacity for the CLI to output either a
text table or a json blob to the terminal. Table is
the default behavior, json is toggled with the --json
flag.
