485 Commits

Author SHA1 Message Date
Keepers
960e8b79a0
cetnralize restoreConfig (#3563)
centralizes all restore configuration management within a restoreConfig struct.  This struct is owned by the control package, which allows it to be utilized by both CLI and SDK consumers.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-12 23:33:22 +00:00
Keepers
cbbc8d2f6c
drive-focused api refactoring prep (#3471)
setting up the drive api calls into files/spaces that will cascade naturally to the addition of an api client for users and sites.  contains some partial implementation of these clients, which will get completed in the next pr.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-02 20:16:47 +00:00
Abin Simon
5c4d57b416
Parallelize restores within a collection for OneDrive (#3492)
This should massively speed up when restoring a collection with many
items. Will not impact much if we have a lot of collections with few
items each.

Numbers 🔢 :
- Restoring ~7000 files, mostly small, totaling 1.5GB
  - Sequential: ~70m
  - Parallel: ~50m
- Restoring 1200 50mb files
  - Sequential: 4h 45m
  - Parallel: <40m

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3011
* closes https://github.com/alcionai/corso/issues/3536

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-02 04:56:11 +00:00
Keepers
cdf26b7988
refactor exchange restore to use interfaces (#3456)
refactors exchange restore from near-duplicate per-category functions and switch-based process trees with interfaces.

At the top of restoring all collections, each category creates a categoryRestoreHandler to supply the necessary restore behavior.  The appropriate handler gets passed in to the collection restore, and all restore code after that takes a single path using a common restore interface to switch between categorical behavior.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-02 00:58:09 +00:00
Abin Simon
6b7502bcb8
Ensure calling Details() twice does not duplicate folder names (#3556)
Related to https://github.com/alcionai/corso/pull/3140 . We were editing the deets in place and returning a pointer. This is fine as long as we were only calling it once, but https://github.com/alcionai/corso/pull/3480 created an additional call for this which causes the `konwnFolders` to be appended twice.

Old backups will still have incorrect data, but an incremental backup on top of it should fix up the data.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Fixes https://github.com/alcionai/corso/issues/3555

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-06-01 18:48:37 +00:00
neha_gupta
06b04c6eff
update files count and total size on CLI for backup (#3480)
<!-- PR description-->
Currently the backup output show,
- total backup size - size of files + size of metafiles
- no of files - no of files + no of metafile
With this change the output of backup command shows size of only files and count include only file count.

NOTE: all current three services results will be impacted here.

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3304

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-05-31 18:02:38 +00:00
Abin Simon
8d68bacfb7
Do not print all the restored items if there are a lot of them (#3495)
For large restore, it just ends up printing a lot of text. This change makes it print a summary instead. We still retain printing the full content if the user asks for a json output as we assume they would be piping it somewhere.

```
Connecting to repository:      1s done
Restoring to folder Corso_Restore_24-May-2023_11-49-44
Connecting to M365:         done
Restoring ∙ 7ceb8e03-bdc5-4509-a136-457526165ec0
Discovered 3 items in backup d5c07808-545d-4906-9ade-f224ca8f13de to restore
Enumerating items in repository:      0s done
Restoring data:     54s done
Restored 304 items.
```

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-31 04:03:23 +00:00
neha_gupta
92845ed139
add user info to sharepoint details (#3509)
<!-- PR description-->

add info about owner to Sharepoint details.

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix

#### Issue(s)

https://github.com/alcionai/corso/issues/3356

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-26 08:35:50 +00:00
Keepers
6e4f30d892
rename ownerName to protectedResourceName (#3513)
#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3478

#### Test Plan

- [x]  Unit test
2023-05-26 01:43:57 +00:00
Keepers
874e3c3a50
cleanup exchange api code, move out of support (#3455)
normalizes exchange api naming and client usage.
Also moves serialization and transformation to and from graph client objects out of graph/support and into the api. Some parts of graph/support have been moved into conn/ exchange instead, since they're more oriented around certain forms of object consumption rather than broad object serialization.

Except for different uses of api client graph servicers, all changes are movement/renaming.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-25 23:53:11 +00:00
Keepers
e8325711b3
update logging standards doc (#3497)
#### Type of change

- [x] 🗺️ Documentation
2023-05-25 06:28:40 +00:00
Keepers
7181e2ef90
move ad-hoc drive api calls into m365/api (#3451)
ensure that all drive-baased graph client calls exist in the m365/api package, not defined ad-hoc throughout the codebase.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-24 21:45:07 +00:00
Keepers
1fb8a426dc
add test name to test context (#3484)
populate a "test_name" clues field in the
tester context, so that logs can be associated
with the calling test.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-24 17:48:36 +00:00
Keepers
7aaabcefdf
check for log file location in artifacts (#3470)
ensure test artifacts have log files

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💚 E2E
2023-05-23 20:21:29 +00:00
Keepers
cf849e9c5d
fix has drive and has mailbox conditions (#3473)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-23 18:08:34 +00:00
ashmrtn
15ef7d5e9b
Try to catch concurrent serialization issues (#3479)
Run a bunch of serialization goroutines at the same time to try to catch any buffer reuse errors there may be. Fails on the old version of corso code that didn't make defensive copies.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-23 16:55:28 +00:00
Keepers
d3611db059
add ownerName to backup list json (#3481)
#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3748

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-05-23 16:31:52 +00:00
Keepers
3d170c5b66
follow-up refinements to m365/api (#3461)
some renaming, mostly refactoring the configuration (select and header declaration for each query) by combining the consts.go and query_param.go file into config.go, and making a more composable set of funcs and consts.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-22 18:37:01 +00:00
jules
11752d869c
Should not error out on "My site not found" (#3469)
This code should not error out on `Unable to retrieve user's mysite URL`.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-22 17:54:44 +00:00
neha_gupta
8bb9df8e75
get repoID from config for events (#3449)
<!-- PR description-->

- Get repoID from config for events for uniformity across repo
- Set a default value if repoID is nil/blank

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3388

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-22 15:04:56 +00:00
ashmrtn
f28c6e53d4
Add tests for older versions of backup details data (#3454)
Add tests for older backup details versions
including things like:
* SharePoint using OneDriveItem type
* not having LocationRef
* not having ItemRef
* folder names in RepoRef
* file names in RepoRef

Recommend viewing with ignore whitespace changes
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3269

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-20 01:41:37 +00:00
ashmrtn
9520de1a37
More robust folder fetching for container cache (#3464)
Update folder cache population
* try to fetch missing parent folders
* refresh current folder if parent wasn't cached
* only populate paths during populatePaths not during IDToFolder

Manually tested email folder resolver population with
stable reproducer and succeeded

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-20 00:05:52 +00:00
jules
7057d94e35
Adding two new apis to corso for validating a user has exchange or one drive is enabled (#3447)
Adds two new apis that expose calls to check whether a user has one drive or exchange enabled

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-18 20:25:32 +00:00
Keepers
6b7745745a
refactor api options (#3428)
now that exchange api has been folded in with the rest of the m365 api, it doesn't make sense to maintain an options file with only exchange functionality.  Since all calls in the file were used 1:1 with some api func, those options have been moved into their respective api funcs.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x] 💚 E2E
2023-05-18 18:49:44 +00:00
Keepers
e64f93a7d3
migrate exchange api to pkg/services/m365 (#3416)
No logic changes, just code movement.

Begins the process ofcentralizing the graph package in pkg/services/m365, instead of dividing up the funcs between different connector subpackages.  While the ownership of apis in each connector subpackage does make sense, this movement will reduce duplication, centralize import arrangements, and make it easy
to export usage of the api to sdk consumers.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #1996

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-17 00:21:20 +00:00
ashmrtn
089a96d437
Create CLI command for maintenance (#3226)
Creates hidden CLI command to run
maintenance.

Flag names and descriptions can be
updated if something else would fit
better

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3077

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-16 21:03:06 +00:00
Keepers
666813865e
case sensitive selection for item IDs (#3397)
Use a strictEquals matcher for item selection when
not matching on item names (no name match assumes
canonical system id match, which is case sensitive).

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3313

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-16 18:15:18 +00:00
Abin Simon
67e055360d
Bump msgraph deps to 1.0.0 (#3345)
Bumps ms graph and all dependent libs to 1.0.0

*Still can't drop betasdk as `models.SitePageable` is still not available.*

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Closes https://github.com/alcionai/corso/pull/3327
* Closes https://github.com/alcionai/corso/pull/3326
* Closes https://github.com/alcionai/corso/pull/3324
* Closes https://github.com/alcionai/corso/pull/3323
* Closes https://github.com/alcionai/corso/pull/3322
* Closes https://github.com/alcionai/corso/pull/3321
* Closes https://github.com/alcionai/corso/pull/3320
* Closes https://github.com/alcionai/corso/issues/3354

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-16 17:44:16 +00:00
Keepers
6febc3ce5b
add namespace to fault items (#3415)
prevent fault items and skips from clobbering eachother by adding a namespace to the fault item that defines a deduplication boundary.  This allows multiple items in a single service operation to have identical ids so long as their namespace differs.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix
- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3283

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-16 17:18:51 +00:00
Keepers
5ca595b7b0
filters cleanup prep for case handling (#3395)
Some tidying up in filters before adding a case specific comparator.  Goals include:
* better test handling/coverage
* swap iota for string in comparator type
* more organized param normalization
* reduction in code duplication

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3313

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-16 16:41:39 +00:00
ashmrtn
04a30ac662
Don't display folders in restore output (#3414)
#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3281

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-15 23:28:04 +00:00
Keepers
3c26b09d71
separate log level and format config (#3398)
Separates the log level and log format.  This will allows sdk consumers using `SeedLogger` to utilize JSON
format logs without sacrificing log level controls.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💪 Manual
2023-05-15 22:03:37 +00:00
Keepers
97ed97c1c3
logic change for sharepoint permission support (#3333)
adds logical changes required for sharepoint permission handling, both backup and restore.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3135

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-15 21:03:00 +00:00
Keepers
a9918d2f78
add repoRef pii transformer (#3383)
adds a transformer func in path/elements.go to safely log a repoRef (or any other dir ref) without needing to conceal the entire string.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
2023-05-12 21:43:16 +00:00
Abin Simon
5995e7aff6
Fix json tag for modified time in sharepoint info (#3353)
Small typo fix for metadata store.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-12 17:25:40 +00:00
Abin Simon
f8aa37b822
Add non delta pagers to exchange (#3212)
When the user's mailbox is full, we cannot make use of delta apis. This adds initial changes needed to create separate delta and non delta pagers for all of exchange.

*I would suggest looking commit wise when reviewing the PR.*

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-12 15:56:49 +00:00
Keepers
1b417af5bb
minor updates to the path.Path interface (#3384)
Swaps path.Append from a single item method to accept a variadic list of string elements.  Since 95% of all calls to path.Append were items, also adds a shorthand
AppendItem func to the interface for easy clarity. Finally, adds a Last() method to elements for getting the last element in the slice.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-12 01:29:17 +00:00
Keepers
6c410c298c
consts for drives, root: (#3385)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 22:46:51 +00:00
Keepers
f2f76d932d
release the sensitive-info flag (#3369)
Not 100% happy with the flag name, and am open to suggestions.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature
- [x] 🗺️ Documentation

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-05-11 04:35:53 +00:00
Keepers
e7d2aeac5d
add drive-level tombstone for deleted drives (#3381)
adds a tombstone collection for any drive that has been completely deleted (or that surfaced from a prior backup, but does not exist in the current) from the driveish account.
---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3379

#### Test Plan

- [x]  Unit test
2023-05-11 03:20:28 +00:00
ashmrtn
2e4fc71310
Add restore path generation code (#3362)
In preparation for switching to folder IDs,
add logic to generate the restore path based
on prefix information from the RepoRef and
LocationRef of items

Contains fallback code (and tests) to handle
older details versions that may not have had
LocationRef

Manually tested restore from old backup that
didn't have any LocationRef information

Manually tested restore checking that
calendar names are shown instead of IDs in
progress bar

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3197
* fixes #3218

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-11 02:49:32 +00:00
Keepers
c5b388a721
add indeets test helper, implement in exchange op (#3295)
Adds a helper for building expected details entries and checking
them after a backup.  Implements the helper in the exchange backup
tests in operations/backup integration.  Will follow with a onedrive
implementation.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3240

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-05-11 02:28:18 +00:00
Keepers
833216c8ae
normalize folders restored in testing (#3363)
Normalize all test folders to use a constant prefix: Corso_Test.  This cleans up and centralizes all per- test variations on the restore destination.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-11 01:02:30 +00:00
ashmrtn
b508ea3b72
Update kopia restore logic to take a (RepoRef, Collection Restore Path) pair (#3337)
Begin expanding the restore logic to take a pair of
paths, one denoting the precise location of the item
in kopia and the other denoting the "restore location"
or path the item should be placed at during restore

This PR is not expected to change system functionality
at all

This is the first of 2 PRs to setup all the logic for
this. This PR does not handle properly merging
together multiple collections that have the same
restore location but different RepoRefs due to recent
updates to the kopia wrapper restore logic

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3197

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-10 01:50:38 +00:00
Vaibhav Kamra
66103892c5
Allow specifying a role to assume when initializing a storage provider (#3284)
Allows caller to specify a IAM role to assume in the Kopia storage provider

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2106 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-05-09 21:40:10 +00:00
Keepers
e67e5be977
remove cli support for exchange item msoft id lookup (#3316)
removes the microsoft item ID from the exchange pathValues set when using only-name selector configuration.  This is the standard config for the cli, thus this removes the ability to filter exchange items from the cli by using their m365 id.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3313

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-05-05 20:28:14 +00:00
ashmrtn
67d5c53420
Fix NPE in SharePoint incremental backup (#3307)
Original code was switching based on the ItemType, but SharePoint historically used the OneDriveItem ItemType, making the system think it should be updating a OneDriveItemInfo struct instead of a SharePointItemInfo struct.

Also add a regression test for older backup formats.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3306

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-04 17:35:12 +00:00
Keepers
7326730e0d
move onedrive.MetadataItem to metadata.Item (#3235)
In keeping with other changes that migrate shared metadata to the onedrive/metadata pkg for exported access.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3135

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-03 20:27:14 +00:00
Keepers
c152ce8b25
move and rename common/time.go (#3215)
moves common/time.go to common/dttm/dttm.go, and renames many of the consts and funcs with names that better explain their design and intent of usage.

Only code movement, no logical changes.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-05-02 19:39:29 +00:00
ashmrtn
8cca7f12df
Create basic maintenance operation (#3225)
Add a maintenance operation to run
kopia maintenance

Using this instead of calling kopia
directly will allow us to hook into
metrics reporting in a more
consistent manner if we want metrics

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3077

#### Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-02 18:18:40 +00:00