14 Commits

Author SHA1 Message Date
Keepers
b94e5a677d
add test.t and pfx to test tree generators (#4777)
this is needed to standardize the presence of a path prefix in all test-helper trees, so that we can use standard test factory helpers for producing complete post-process data.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-12 17:44:23 +00:00
Keepers
8af0e31e4e
remove 'errors' from count keys (#4813)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
2023-12-08 19:45:39 +00:00
Keepers
54ba241fbe
require rootID on tree construction (#4746)
Turns out the root ID name isn't an appropriate match for establishing the root node.  Instead, the backup hander is now extended with a getRootFolder method and will pass the expected root folder ID into the tree's constructor func to ensure we establish the correct root node.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-12-07 23:54:45 +00:00
Abhishek Pandey
3ee1a8d166
Minor url cache logging & counting improvements (#4801)
<!-- PR description-->

1. Move cache miss logs to debug, as it can be chatty sometimes. We will leverage the end of operation `URLCacheMiss` counter to get this information instead.
2. Counter renaming. 
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-12-05 02:18:13 +00:00
Keepers
6e572fd133
add and delete file support in the tree (#4724)
now that folder handling is complete, we can ingest items in the delta tree as well.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-12-04 16:37:53 +00:00
Keepers
d9c42f790c
introduce flat deletions to delta tree (#4719)
there are certain enumeration cases where tombstoning isn't the correct behavior, and we want to delete a folder from the tree entirely.  This primarily occurrs when we have a create->delete pair of markers either within or across enumerations.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🌻 Feature

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-12-01 00:12:57 +00:00
Keepers
6aff258c8b
add boilerplate for drive tree processing (#4716)
begins implementation of the drive delta tree support by adding boilerplate funcs for getting the backup data, and adding a framework of unit tests that will be used bring testing parity up to par with current tests.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Issue(s)

* #4689

#### Test Plan

- [x]  Unit test
2023-11-28 21:07:49 +00:00
Keepers
f4ed4d7250
bump xsync to v3 (#4704)
three changes
1. bumps the xsync package to v3
2. creates a common package for synced maps
3. replaces all xsync MapOf imports with the new common/syncd package.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-17 18:57:29 +00:00
Keepers
9ecefd2569
use counter in exchange backup enum (#4661)
Increase proliferation of the count bus to record runtime stats.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-11-16 23:22:00 +00:00
Keepers
5e0014307c
remove ctx embedding for counter (#4497)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-31 16:15:05 +00:00
Keepers
5cc68e27dc
use count bus in kopia backups (#4482)
uses the count bus in the kopia backup package.
This currently duplicates counts that we're getting
from the kopia stats.  A later pr will remove the old
stats entirely in favor of the counter.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-10-23 23:29:56 +00:00
Keepers
6bc40ea2aa
add throttling, token counters to operation (#4318)
Adds context embedding to pkg/count, and embeds
the backup, restore, and export operation ctx with the operations count.bus.  The bus is extracted from the ctx in api middleware to track throttling response counts, as well as token consumption counts.  Counts are globally tracked, not on a per-time or per resource basis.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-09-22 00:43:16 +00:00
Keepers
5a78f478a1
add operations test for adv rest (#3783)
ads operations level tests for advanced restore
configuration on all three services.
Code is largely boilerplate between each service, but with just enough quirks that full consolidation would require excess jumping through hoops.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3562

#### Test Plan

- [x] 💚 E2E
2023-07-17 03:56:08 +00:00
Keepers
2a150cc610
add counter for skip-collision counting (#3722)
Introduces a counting bus, and threads it into restore operations so that we can count the number of
collision skips that occur.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-06 19:22:02 +00:00