first step towards having a centralized set of error sentinels that can be passed around corso instead of re-using low level error sentinels like those found in the graph/errors.go file.
This PR works as a standalone, and only handles the lowest hanging fruit. SDK consumers will need to change their error enum references, but all api behavior should remain the same.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #4685
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Part 1 of two changes I've been carting around since the hackathon.
This PR moves the /cli/config package into /pkg/config. This allows
sdk consumers to access corso configuration handling, intead of
requiring the CLI package to access it.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Switch restore code path to use service-level handlers. Does a few
things:
* switches existing service-level functions to be methods on
service-level handlers
* update interfaces as necessary
* moves some logic from old controller-level restore function to either
the new handlers or the operation-level function
* removes old code
May be easiest to review by commit
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4254
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Rework restore return status so that later PRs will have a smaller diff
Instead of returning a status and then waiting on the message just return the restore stats directly. The status getter was only setup to wait for one item anyway and was setup to wait after the entire restore operation already completed (at end of
m365/restore.go:ConsumeRestoreCollections())
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #4254
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Increase proliferation of the count bus to record runtime stats.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
This commit include a few things that improves the progresbars and other info messages are printed out during the CLI run.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
this is an alternative possible response from graph api when the client is updating the token in-flight. We normally catch this for item downloads, but are defaulting on permission downloads.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🐛 Bugfix
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
replace all cases of new|wrap|stack(...).WithClues(ctx) with the flattened new|wrap|stackWC(ctx, ...) functions introduced in the latest clues bump.
Other changes added:
* remove WithClues builders when the error producer already called it.
* corrected some usages of ictx within loops.
no logic changes, just cleanup.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
1. moves the m365/graph package from internal to pkg/services/api so that options are accessible to sdk users.
2. exposes graph.Options in the api client.Service call.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
To assist in debugging via logs, add the set of
categories an operation is acting on to the
clues set in the context
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
remove start events from all operations
#### Does this PR need a docs update or release note?
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/4440
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
hands the backup resource into the drive collection for the handler to use to record as the siteID
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #3988
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Adds context embedding to pkg/count, and embeds
the backup, restore, and export operation ctx with the operations count.bus. The bus is extracted from the ctx in api middleware to track throttling response counts, as well as token consumption counts. Counts are globally tracked, not on a per-time or per resource basis.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Still a few pending items like CLI handling and restoring specific items. Will be handled in a follow up.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3992
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
1. Renamed `IsBackupRunnable` to `IsServiceEnabled` & extended to restore operation. Removed `checkServiceEnabled`. Reasons:
- The above 2 functions were doing the same thing. Removed `checkServiceEnabled` in favor of `IsBackupRunnable` since we want this check to be as soon as possible during backup/restore op initialization
- Renamed `IsBackupRunnable` to `IsServiceEnabled`, because a) we are only doing service enabled checks right now, b) common code that can be used for both restore & backup.
2. Wire corso code to use new helpers in internal/m365/common.go
3. Note: SDK wiring and related integ tests will be added in a follow up PR
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3844
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Remove references to the kopia package from
`pkg/store` package so that kopia can import
that package itself. Do this by using
interfaces where needed in `pkg/store`
instead of concrete struct types
These changes will make cleaning up
incomplete backups a little neater since
that code will need to lookup both
manifests and backup models
This PR is just minor renaming and fixups,
no logic changes
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3217
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
Only code movement, no code changes.
Moved services to `/internal/m365/services/{onedrive,sharepoint,exchange}`
Moved collections to `/internal/m365/collection/{drive,site}`
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
run `go get -u` to update all packages.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
If the restore configuration specifies a protected resource as a restore target, use that as the destination for the restore. First step is to ensure the provided target can be retrieved and identified.
---
#### Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
Adds api handlers for creating document libraries in sharepoint. This is the first step in allowing us to restore drives that were deleted between backup and restore.
---
#### Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
ads operations level tests for advanced restore
configuration on all three services.
Code is largely boilerplate between each service, but with just enough quirks that full consolidation would require excess jumping through hoops.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #3562
#### Test Plan
- [x] 💚 E2E
a variety of small updates that came
from manual testing of restore with various
collision and destination combinations.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #3562
#### Test Plan
- [x] 💪 Manual
Introduces a counting bus, and threads it into restore operations so that we can count the number of
collision skips that occur.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
In case of Onedrive Restore give a proper message if the path provided do not exist or restore is done on empty backup.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3579
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
Adds a func that normalizes the restore config in case inputs don't match accepted or expected values.
Also moves DefaultTestRestoreConfig out of /tester and into /control/testdata to avoid circular dependency issues.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
renames /internal/connector to /internal/m365. No logic changes in this PR. Only the dir rename, import renames, and one linter shadowing rename.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1996
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
centralizes all restore configuration management within a restoreConfig struct. This struct is owned by the control package, which allows it to be utilized by both CLI and SDK consumers.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
setting up the drive api calls into files/spaces that will cascade naturally to the addition of an api client for users and sites. contains some partial implementation of these clients, which will get completed in the next pr.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1996
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
This should massively speed up when restoring a collection with many
items. Will not impact much if we have a lot of collections with few
items each.
Numbers 🔢 :
- Restoring ~7000 files, mostly small, totaling 1.5GB
- Sequential: ~70m
- Parallel: ~50m
- Restoring 1200 50mb files
- Sequential: 4h 45m
- Parallel: <40m
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3011
* closes https://github.com/alcionai/corso/issues/3536
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
adds logical changes required for sharepoint permission handling, both backup and restore.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3135
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
No logic changes, just renaming and minor cleanup.
PRs to follow:
1. collect various maps in onedrive collections into a single cache.
2. logic changes and tests to produce sharepoint permissions backup/restore e2e.
3. extend permission identity-type retention.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3135
#### Test Plan
- [x] ⚡ Unit test
move the end-of-op notification event to a defer within backup/restore.Do(). This ensures we'll get the end event data even in case of failure.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #3388
#### Test Plan
In preparation for switching to folder IDs,
add logic to generate the restore path based
on prefix information from the RepoRef and
LocationRef of items
Contains fallback code (and tests) to handle
older details versions that may not have had
LocationRef
Manually tested restore from old backup that
didn't have any LocationRef information
Manually tested restore checking that
calendar names are shown instead of IDs in
progress bar
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3197
* fixes#3218
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Begin expanding the restore logic to take a pair of
paths, one denoting the precise location of the item
in kopia and the other denoting the "restore location"
or path the item should be placed at during restore
This PR is not expected to change system functionality
at all
This is the first of 2 PRs to setup all the logic for
this. This PR does not handle properly merging
together multiple collections that have the same
restore location but different RepoRefs due to recent
updates to the kopia wrapper restore logic
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3197
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
moves common/time.go to common/dttm/dttm.go, and renames many of the consts and funcs with names that better explain their design and intent of usage.
Only code movement, no logical changes.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
The fetch paralellism checks and logs occur on
every item streamed from GC. This is a bit chatty,
and has been moved upstream in the process for
a more centralized behavior.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
Begins hiding PII with clues.Hide injections in our structured logging aggregators. This is a Non-
comprehensive change, and improvements here will be ongoing.
---
#### Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #2024
#### Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
While fail-fast and best-effort make for good
categories at the extreme ends of error handling,
we keep finding outselves wanting to operate
in a middle ground. This change introduces a new
error handling category: FailAfterRecovery. This
option tells corso to complete as much of its
process as it can, even if it recovers from errors.
But at the end of processing, if it recovered from
any errors, an error is returned for the operation.
This behavior is the new failure handling default,
instead of failFast.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
The observe PII handlers are getting replaced by
the more standardized clues secrets.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2024
#### Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
Replaces the operations graphConnector reference
with an interface. Restore and Backups have
separate, unique interfaces.
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2825
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Moves the generation of graphConnector to the
NewBackupOp and NewRestoreOp constructors
inside the repository. This removes gc creation
from the backup and restore operations, and
requires a reference to gc to exist in operations.
This sets up two changes: 1/ mocking of GC
within operation (the next PR will replace it with
an interface). 2/ the ability to perform a gc
validation step within the operation construction,
and thus the capacity to look up a resource owner
display name or id, and vice versa.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2825
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Updates and corrects input aliasing according to
the following rules (in priority order):
1. if the library name is usable, use it
2. if not, alias to the package name
3. if the package name is weird, alias sensibly
4. in case of collision, alias more distant imports
5. aliases should be consistent throughout
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
Adds a logger-oriented metrics tracker to aggregate statistics like requests-per-minute and average
http request time.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #2212
#### Test Plan
- [x] 💪 Manual
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.
Might follow up with a linter to enforce this change.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Bugfix:
Sharepoint LibraryFolder scopes always return
zero matches in filtering with a prefix due to the
path not having the drive root prefix truncated, as is
expected of onedrive-based storage paths.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🐛 Bugfix
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
Moves the drive fetcher and pagers into the
onedrive/api package so that it can be easily re-
used by other packages. Also moves the mocks
out from local testing and into a mock package
nested in the api pkg.
first of a couple PRs fixing up sharepoint testing
issues.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Test Plan
- [x] 💚 E2E
changes a couple things in the backup cli output:
* appends the backup id after completion
* prints an error if the backup fails
* replaces multierr with a clues error
* logs each backup failure individually
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] 💪 Manual
