336 Commits

Author SHA1 Message Date
neha_gupta
09f3196ee0
Onedrive restore- handle unknown path error (#3643)
<!-- PR description-->
 In case of Onedrive Restore give a proper message if the path provided do not exist or restore is done on empty backup.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3579

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-07-04 11:37:16 +00:00
Keepers
8b81728488
fix concurrency limiter init race (#3715)
initializes the concurrency limiter as part of CLI pre-run initialization to ensure we don't accidentally skip out on including the limiter in middleware.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3695

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-30 23:54:54 +00:00
Keepers
5949a3bcd7
add logging indicating full backups (#3721)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3720
2023-06-30 23:02:01 +00:00
Keepers
43d4a4d5e8
remove top query param in paged queries (#3719)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-30 04:49:14 +00:00
Keepers
10c8fd9d12
adds test coverage for events in exchange incrementals (#3661)
We still have events excluded from the exchange
incrementals integration tests.  For future safety, these need to be part of the testing group.

However, there's a hitch.  Our primary test user is broken again, and cannot retrieve any calendars
except for the well-known set: calendars, birthdays, holidays.  We'll have to fix that before merging.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💚 E2E
2023-06-27 20:05:25 +00:00
ashmrtn
9cbd46aabe
Add more subtests to aid in debugging (#3667)
Pass around context more and add more subtests so that the logs actually show what's happening.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3666

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-06-26 23:49:13 +00:00
Keepers
6b002a1684
exclude flaky integration test assertions (#3660)
Comments out some particularly flaky test assertions. We have other assurances around this data already, so the exclusion is relatively safe.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💚 E2E
2023-06-23 22:34:34 +00:00
Abin Simon
b0305a5319
Update attachments for events which drift from series master (#3644)
This updates any changes to attachments for individual event instances
for the ones that differ from the series master.<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2835

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-06-22 13:37:05 +00:00
Abin Simon
9199db8f45
Handle event exceptions in caledar (#3589)
This ensures that we backup and restore and exceptions in the recurring events. One thing pending here is fixing up the attachments. I hope to create that as a separate PR. This can probably go in separately as well.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2835

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-19 11:57:32 +00:00
Keepers
416383a99c
cascade restoreCfg collision policy into onedrive (#3623)
Adds collision policy handling to onedrive item posts.  This allows us to override the default "replace" behavior that currently returns a 409 for the creation endpoint, in case we want to use skip (ie: fail) or copy handling.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-16 00:07:09 +00:00
ashmrtn
c0f428ddc8
Refactor backup code to use BackupBases functions (#3596)
Now that BackupBases defines functions, leverage them
in other code to reduce the number of times we fetch
Backup models and leverage the stronger invariants
the new FindBases function has

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3525

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-06-15 17:43:18 +00:00
Keepers
46eedc0ad1
add default restore config normalizer (#3613)
Adds a func that normalizes the restore config in case inputs don't match accepted or expected values.

Also moves DefaultTestRestoreConfig out of /tester and into /control/testdata to avoid circular dependency issues.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
2023-06-15 16:04:42 +00:00
Keepers
ce72acbcc1
auto-log recoverable errors with stack (#3598)
automatically log when we add a recoverable error or a skipped item to fault.  This log will include a stack trace of the call from the location of the logged recoverable.  Clues does not have a method for pulling a stack trace out of an error yet; that can be added at a future date.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-13 22:18:18 +00:00
Keepers
2f6d731993
rename connector -> m365 (#3600)
renames /internal/connector to /internal/m365.  No logic changes in this PR.  Only the dir rename, import renames, and one linter shadowing rename.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-13 18:35:39 +00:00
Keepers
960e8b79a0
cetnralize restoreConfig (#3563)
centralizes all restore configuration management within a restoreConfig struct.  This struct is owned by the control package, which allows it to be utilized by both CLI and SDK consumers.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-12 23:33:22 +00:00
Abin Simon
7d1d9295eb
Read items within a collection from kopia on demand (#3506)
Don't read all the files that we need to restore, just the collections in the beginning. The rest of them are read using Fetch on demand when they will be restored.

I haven't tested for any changes in memory consumption, but this brings down time taken for "Enumerating items in repository" to <5s even for huge number of files.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3011
* closes https://github.com/alcionai/corso/issues/3440
* closes https://github.com/alcionai/corso/issues/3537

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-06-05 11:02:39 +00:00
Keepers
cbbc8d2f6c
drive-focused api refactoring prep (#3471)
setting up the drive api calls into files/spaces that will cascade naturally to the addition of an api client for users and sites.  contains some partial implementation of these clients, which will get completed in the next pr.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-02 20:16:47 +00:00
Abin Simon
5c4d57b416
Parallelize restores within a collection for OneDrive (#3492)
This should massively speed up when restoring a collection with many
items. Will not impact much if we have a lot of collections with few
items each.

Numbers 🔢 :
- Restoring ~7000 files, mostly small, totaling 1.5GB
  - Sequential: ~70m
  - Parallel: ~50m
- Restoring 1200 50mb files
  - Sequential: 4h 45m
  - Parallel: <40m

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3011
* closes https://github.com/alcionai/corso/issues/3536

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-02 04:56:11 +00:00
Keepers
cdf26b7988
refactor exchange restore to use interfaces (#3456)
refactors exchange restore from near-duplicate per-category functions and switch-based process trees with interfaces.

At the top of restoring all collections, each category creates a categoryRestoreHandler to supply the necessary restore behavior.  The appropriate handler gets passed in to the collection restore, and all restore code after that takes a single path using a common restore interface to switch between categorical behavior.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-02 00:58:09 +00:00
ashmrtn
6fd453a699
Remove old code for finding bases (#3526)
Now that we're finding bases with a different component
remove the code that used to find bases since it's no
longer in use

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3202

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-01 15:41:16 +00:00
Keepers
2da554f00e
minor logging updates (#3546)
Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
2023-06-01 07:01:08 +00:00
ashmrtn
7cb01b1e27
Wrap new base finder and wire it up (#3524)
Take the new base finder code, wrap it in
a thin wrapper and use it in place of the
previous way of finding base snapshots

This solution is not focused on efficiency
as it does result in looking up backup
models multiple times. It's more to get the
new way of finding bases out and then we
can go back and smooth things over when
we have the chance

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3202

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-31 18:27:15 +00:00
neha_gupta
06b04c6eff
update files count and total size on CLI for backup (#3480)
<!-- PR description-->
Currently the backup output show,
- total backup size - size of files + size of metafiles
- no of files - no of files + no of metafile
With this change the output of backup command shows size of only files and count include only file count.

NOTE: all current three services results will be impacted here.

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3304

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-05-31 18:02:38 +00:00
neha_gupta
bb16dea4c2
don't throw error on backup with service disabled (#3426)
<!-- PR description-->

If a user does not have a service(like exchange) enabled- don't throw an error but just skip that user and move forward with other backups.

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3392

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-05-29 06:24:15 +00:00
ashmrtn
9675c54c46
Add new code for finding bases for incremental backups (#3502)
Add a new set of functions to find
bases for incremental backups. The
new set of functions return
* set of backup models and the reasons for selecting each
* set of snapshots to merge and the reasons for selecting each
* set of snapshots for kopia-assisted incrementals and reasons for selecting each

New code is not wired into existing
system so no behavioral changes are
expected

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3202

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-27 02:56:06 +00:00
ashmrtn
264c1968ab
Move some interfaces to kopia package (#3500)
Searching for new bases is mostly kopia-specific but is going to require the store interface. With the way packages and interfaces are currently laid out it makes it impossible to easily get the interface added to the inject package without a cycle.

This moves functions that the kopia package provides into a subpackage of the kopia package that other things can then import

This change only moves code. No logic is updated

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3202

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-26 22:22:49 +00:00
Keepers
874e3c3a50
cleanup exchange api code, move out of support (#3455)
normalizes exchange api naming and client usage.
Also moves serialization and transformation to and from graph client objects out of graph/support and into the api. Some parts of graph/support have been moved into conn/ exchange instead, since they're more oriented around certain forms of object consumption rather than broad object serialization.

Except for different uses of api client graph servicers, all changes are movement/renaming.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-25 23:53:11 +00:00
Keepers
7181e2ef90
move ad-hoc drive api calls into m365/api (#3451)
ensure that all drive-baased graph client calls exist in the m365/api package, not defined ad-hoc throughout the codebase.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-24 21:45:07 +00:00
Keepers
1fb8a426dc
add test name to test context (#3484)
populate a "test_name" clues field in the
tester context, so that logs can be associated
with the calling test.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-24 17:48:36 +00:00
Keepers
e64f93a7d3
migrate exchange api to pkg/services/m365 (#3416)
No logic changes, just code movement.

Begins the process ofcentralizing the graph package in pkg/services/m365, instead of dividing up the funcs between different connector subpackages.  While the ownership of apis in each connector subpackage does make sense, this movement will reduce duplication, centralize import arrangements, and make it easy
to export usage of the api to sdk consumers.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-17 00:21:20 +00:00
ashmrtn
089a96d437
Create CLI command for maintenance (#3226)
Creates hidden CLI command to run
maintenance.

Flag names and descriptions can be
updated if something else would fit
better

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3077

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-16 21:03:06 +00:00
Abin Simon
67e055360d
Bump msgraph deps to 1.0.0 (#3345)
Bumps ms graph and all dependent libs to 1.0.0

*Still can't drop betasdk as `models.SitePageable` is still not available.*

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Closes https://github.com/alcionai/corso/pull/3327
* Closes https://github.com/alcionai/corso/pull/3326
* Closes https://github.com/alcionai/corso/pull/3324
* Closes https://github.com/alcionai/corso/pull/3323
* Closes https://github.com/alcionai/corso/pull/3322
* Closes https://github.com/alcionai/corso/pull/3321
* Closes https://github.com/alcionai/corso/pull/3320
* Closes https://github.com/alcionai/corso/issues/3354

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-16 17:44:16 +00:00
Keepers
97ed97c1c3
logic change for sharepoint permission support (#3333)
adds logical changes required for sharepoint permission handling, both backup and restore.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3135

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-15 21:03:00 +00:00
Keepers
60f6d4a035
renaming/cleanup for sharepoint perms pt.1 (#3330)
No logic changes, just renaming and minor cleanup.

PRs to follow:
1. collect various maps in onedrive collections into a single cache.
2. logic changes and tests to produce sharepoint permissions backup/restore e2e.
3. extend permission identity-type retention.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3135

#### Test Plan

- [x]  Unit test
2023-05-12 21:23:52 +00:00
Keepers
695f8060da
insert restoreEnd event into deferred func (#3400)
move the end-of-op notification event to a defer within backup/restore.Do(). This ensures we'll get the end event data even in case of failure.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3388

#### Test Plan
2023-05-12 19:18:53 +00:00
Abin Simon
f8aa37b822
Add non delta pagers to exchange (#3212)
When the user's mailbox is full, we cannot make use of delta apis. This adds initial changes needed to create separate delta and non delta pagers for all of exchange.

*I would suggest looking commit wise when reviewing the PR.*

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-12 15:56:49 +00:00
Keepers
1b417af5bb
minor updates to the path.Path interface (#3384)
Swaps path.Append from a single item method to accept a variadic list of string elements.  Since 95% of all calls to path.Append were items, also adds a shorthand
AppendItem func to the interface for easy clarity. Finally, adds a Last() method to elements for getting the last element in the slice.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-12 01:29:17 +00:00
ashmrtn
674d3eec91
Switch to folder IDs in Exchange (#3373)
Store all folders from Exchange by folder ID
in kopia. Remove the old duplicate folder
name stop-gap measure as well since it's no
longer required

Update tests to check LocationPath instead
of FullPath since LocationPath still has
display name info

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3197

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-05-11 23:07:38 +00:00
Keepers
6c410c298c
consts for drives, root: (#3385)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 22:46:51 +00:00
Keepers
caea3ab6da
make the linter happy (#3394)
#### Type of change

- [x] 🧹 Tech Debt/Cleanup
2023-05-11 20:38:33 +00:00
ashmrtn
2e4fc71310
Add restore path generation code (#3362)
In preparation for switching to folder IDs,
add logic to generate the restore path based
on prefix information from the RepoRef and
LocationRef of items

Contains fallback code (and tests) to handle
older details versions that may not have had
LocationRef

Manually tested restore from old backup that
didn't have any LocationRef information

Manually tested restore checking that
calendar names are shown instead of IDs in
progress bar

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3197
* fixes #3218

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-11 02:49:32 +00:00
Keepers
c5b388a721
add indeets test helper, implement in exchange op (#3295)
Adds a helper for building expected details entries and checking
them after a backup.  Implements the helper in the exchange backup
tests in operations/backup integration.  Will follow with a onedrive
implementation.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3240

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-11 02:28:18 +00:00
Keepers
c0725b9cf9
some quick logging and error naming updates (#3348)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3344

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 02:03:31 +00:00
Keepers
833216c8ae
normalize folders restored in testing (#3363)
Normalize all test folders to use a constant prefix: Corso_Test.  This cleans up and centralizes all per- test variations on the restore destination.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 01:02:30 +00:00
ashmrtn
b508ea3b72
Update kopia restore logic to take a (RepoRef, Collection Restore Path) pair (#3337)
Begin expanding the restore logic to take a pair of
paths, one denoting the precise location of the item
in kopia and the other denoting the "restore location"
or path the item should be placed at during restore

This PR is not expected to change system functionality
at all

This is the first of 2 PRs to setup all the logic for
this. This PR does not handle properly merging
together multiple collections that have the same
restore location but different RepoRefs due to recent
updates to the kopia wrapper restore logic

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3197

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-10 01:50:38 +00:00
Keepers
3a2d0876dd
consolidate aggregation of parent-item exclude map (#3258)
introduces a new type wrapping a nested map so that aggregation of globally excluded items in driveish services don't need to manage the map updates themselves.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2340

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-04 19:32:47 +00:00
Keepers
d067a79483
move permissions into metadata (#3216)
Moves the UserPermissions struct into onedrive/metadata as
the Permission struct, for better shared support across onedrive,
sharepoint, and any other packages that reference it.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3135

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-03 18:52:25 +00:00
Keepers
c152ce8b25
move and rename common/time.go (#3215)
moves common/time.go to common/dttm/dttm.go, and renames many of the consts and funcs with names that better explain their design and intent of usage.

Only code movement, no logical changes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-02 19:39:29 +00:00
ashmrtn
8cca7f12df
Create basic maintenance operation (#3225)
Add a maintenance operation to run
kopia maintenance

Using this instead of calling kopia
directly will allow us to hook into
metrics reporting in a more
consistent manner if we want metrics

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3077

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-02 18:18:40 +00:00
ashmrtn
b6ef2d5266
Create wrapper around kopia maintenance (#3224)
Create a wrapper function around the kopia
maintenance operation. Will allow users to
run maintenance, though they still need to
be careful not to run maintenance
concurrently if they override the username
and hostname that kopia uses

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3077

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-02 00:52:15 +00:00