Adds a logger-oriented metrics tracker to aggregate statistics like requests-per-minute and average
http request time.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Supportability/Tests
#### Issue(s)
* #2212
#### Test Plan
- [x] 💪 Manual
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.
Might follow up with a linter to enforce this change.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Bugfix:
Sharepoint LibraryFolder scopes always return
zero matches in filtering with a prefix due to the
path not having the drive root prefix truncated, as is
expected of onedrive-based storage paths.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🐛 Bugfix
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
Exchange now holds a reference to the parent folder so be sure to update
that if needed when merging backup details
Calendar is the only thing that populates location ref right now so this
alone doesn't completely solve the issue
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2819
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
updating sanity tests: fix the permissions check by not using reflect.DeepEquals, and various code cleanup.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Test
#### Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
This reverts commit 1945db6b65827e6dd6d57de8ac42fe4a119db685.
We do add the `isMeta` flags for missing items at f60207ee32/src/pkg/repository/repository.go (L363)
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
adds a resourceOwnerName property to the op
backup struct. Also adds a resource owner id
and owner name to the backup.backup. These
values will be used for both identification and for
end user display. Values are not currently populated
reliably, nor are they used. This change only
allows them to exist.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2825
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Updates the sharepoint cli examples to better
reflect the current support and utilities.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2786
#### Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
Renames a lot of the names in scopes that contain
the word "filter" with the word "info". The basis here
is to reduce confusion in the libraries. Filter as a term
is overloaded. It currently means: 1/
the set of scopes that comparisons _must_ pass
to succeed; 2/ the pkg and struct that handles
comparison logic; and 3/ the set of scopes that
compare values in details.ItemInfo instead of in
the item's storage or logical path. The third
category is the primary overload here, and is
better off with a different name.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
This option is only consumed by Exchange right now and will control the number of items fetched in parallel per collection. Note that the total number of parallel item fetches may be higher is kopia is concurrently uploading multiple collections.
This PR does not expose this option at the CLI layer
Manually tested to ensure it
* uses the default if options value is 0
* uses the default if options value > default
* uses option value otherwise
There's another PR open for this but it's not expected to main. We still
want to make sure the change goes in though, hence this PR.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
A last-second change in 2708 caused us to
pass along the wrong fault.Errors into backup
persistence, thus slicing the count of skipped
items. That's been fixed, along with improved
end-of-operation logging of fault errors.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #2708
#### Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
Adds a new streamstore controller for fault.Errors. This provides large scale, extensible file storage for fault errors to be persisted, much like we do for backup details.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2708
#### Test Plan
- [x] 💚 E2E
Doesn't seem like we're using these
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Test
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2373
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #2757
#### Test Plan
- [x] ⚡ Unit test
In order to retrieve all clues structured error data in tests, we need to extract it from the error using the clues library.
This change appends `clues.ToCore(err)` to all
variations of `assert.NoError(t, err)`. The only
other changes are those necessary to preserve
linting, or to produce an error variable for the
ToCore call.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🤖 Test
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Treat a 404 as an unavailable item and skip it similar to malware items.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Treat a 404 as an unavailable item and skip it similar to malware items.
This skips the call to `getRepoModel` if metrics are disabled and the repository ID is not required.
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2802
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Change to storing OneDrive files by ID instead of their OneDrive display name. This allows delta token-based incrementals to use an exclude list to remove previously backed up items from a backup during hierarchy merging
Also updates the following:
* selectors to match on file display name instead of kopia file name
* ShortRefs for OneDrive files to update when the file display name is updated
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* closes#1535
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
The selector moniker "libraries" is incorrect. It should refer to the library folder, since it only
matches on directory structures within a given
drive. The 'library' is analogous to the drive
itself, and will need a separate selector of its
own.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #2757
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
This PR adds a helper function that returns a newly introduced `Site` struct that has a site ID and site webURL for a given m365 tenant.
---
#### Does this PR need a docs update or release note?
- [X] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [X] 🌻 Feature
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2777
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [X] ⚡ Unit test
```
--- PASS: TestM365IntegrationSuite (2.49s)
--- PASS: TestM365IntegrationSuite/TestSites (1.82s)
--- PASS: TestM365IntegrationSuite/TestSites/site (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#01 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#02 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#03 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#04 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#05 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#06 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#07 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#08 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#09 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#10 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#11 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#12 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#13 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#14 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#15 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#16 (0.00s)
--- PASS: TestM365IntegrationSuite/TestSites/site#17 (0.00s)
```
Adds a path.Build() func that can arbitrarily build a path according to expected standards without first initializing a path.Builder{}.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Adds the item struct to the fault package for tracking serializable and dedupliatable error sources.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
#### Issue(s)
* #2708
#### Test Plan
- [x] ⚡ Unit test
This adds the missing .dirmeta files which are needed to make sure that permissions are being restored properly.
By adding a dirmeta entry for each folder that is part of the items, we make sure that all folders get their own collection which in turn will help us properly restore permissions for all folders.
**This also makes it so that meta items are not returned from older backups.**
*Although the newer version can fetch permissions file using the Fetch API, we still have to populate the items as it cannot fetch items from previous directories which is needed to make sure we can restore permissions correctly.*
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2697
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- Insert PR description-->
ItemTypes for SharePoint were `SharePointItem` only. This did not allow for data to be separated by category.
Newly created categories:
- SharePointLibrary
- SharePointList
- SharePointPage
Breaks history for `SharePointItems`. All previous SharePoint items are available during `Corso List` command. However, when using the details command with the backup ID for `SharePointItems` restored prior to this PR , the return will be:
```
no items match the specified selectors
```
---
#### Does this PR need a docs update or release note?
- [x] 📝 : Will need documentation
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #2709<issue>
#### Test Plan
- [x] ⚡ Unit test
Have the `In` filter take a slice of values instead
of a joined string. Change negated slice filter behavior
to be the boolean negation instead of
(!filter-val-1 || !filter-val-2 || ...)
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1535
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
With onedrive storage file names being changed from the file display name to the file id, we need a more granular form of indentification when using selectors to choose which values count as matchable fields.
This change modifies the selector PathValues to return slices of strings for each category instead of a single string, and the reducer matches on any. This will allow each service to decide what values are considered equivalent (id, shortRef, a value inside the info, etc) for each property.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2708
#### Test Plan
- [x] ⚡ Unit test
## Description
Now that fault errors is in place, we don't need
the operation stats errs.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Refactors the stream store so that it can comply with multiple different writers and readers beyond just the details.Details struct. This will allow us to use the stream store to ingest large lists of errors fault items without burdening the backup manifest.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2708
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Initial code movement before getting into larger refactor. The goal is to refactor the stream store so that it can comply with multiple different writers and readers beyond just the details.Details struct. This will allow us to use the stream store to ingest large lists of errors fault items without burdening the backup manifest.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2708
#### Test Plan
- [x] ⚡ Unit test
<!-- Insert PR description-->
#### Adds Email recipients to the backup details for Exchange Mail. If the recipients are of length greater than one, the list of individuals is included and separated by a `,`
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2687<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] ⚡ Unit test
<!-- Insert PR description-->
Selector expansion for filters includes the following:
- time flags for details filtering
- testing
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* related to #2602<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] ⚡ Unit test
<!-- Insert PR description-->
Parent Path field added to `Exchange` Backup Details
---
- Only Exchange mail will display the path for backup details.
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes #2352<issue>
#### Test Plan
- [x] ⚡ Unit test
Metadata files don't need to appear in backup
details, which means they also shouldn't be
merged during incremental backups. If we
expect them to be merged then we'll fail the
item count check for the merge process
This excludes items with metadata extensions
from merging by not adding the to the list
of items to merge
Also make it so that metadata items don't
create backup details entries in the first
place
Manually tested to ensure
restore(backup(onedrive)) still works as
expected
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
* closes#2498
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
adds a hidden flag to backup details that
allows the caller to see all details retrieved
by the repository, without running the results
through the selector reduce step.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2675
#### Test Plan
- [x] 💪 Manual
## Description
Don't populate users when GraphConnector instance is created
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1994
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2373
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
While the `isMeta` flag will solve this in the
future, the current state requires us to filter
out all meta and dirmeta files, even historically.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- Insert PR description-->
Updates for `./corso backup details sharepoint --backup <bID>`
- Ensures that Drive Name is part of the backup
- Adds field DisplayName to backup details
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
*closes #2601 <issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
## Description
Leave out account and storage subpackages as those are used by tester and create an import cycle.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2373
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Renaming the funcs in the fault
package to be more clear about
their purpose and behavior. Largely
just find&replace changes, except
for fault.go and the fault examples.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Does this PR need a docs update or release note?
- [ ] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Remove the display and interaction with .meta
files from details entries.
## Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
## Type of change
- [x] 🐛 Bugfix
## Test Plan
- [x] ⚡ Unit test
## Description
Realized we had a race condition: in an async
runtime it's possible for an errs.Err() to be
returned by multiple functions, even though that
Err() was only sourced by one of them. The
addition of a tracker contains the returned
error into the scope of that func so that only
the error produced in the current iteration is
returned.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1970
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #2529
## Test Plan
- [x] ⚡ Unit test
