99 Commits

Author SHA1 Message Date
Keepers
d7839ac607
update cli docs (#3310)
updates and improves cli docs.  in part for easier reading and explanation.  in part to remove all references to resource owner flags in details and restore.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
2023-05-04 20:03:28 +00:00
Keepers
9b21699b6b
reset body reader on corso retry handling (#3280)
The kiota compressor middleware will attempt to compress the request body.  In the event that we have a corso-middleware- retriable response (eg: status 500), we need to reset the seek position of the req.Body, similar to how graph api does in their retry handler, or else the re-run of the compressor will already have read the full req.Body and the retried call will have a zero len body.

---

#### Does this PR need a docs update or release note?

- [y]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
2023-05-04 18:31:12 +00:00
ashmrtn
67d5c53420
Fix NPE in SharePoint incremental backup (#3307)
Original code was switching based on the ItemType, but SharePoint historically used the OneDriveItem ItemType, making the system think it should be updating a OneDriveItemInfo struct instead of a SharePointItemInfo struct.

Also add a regression test for older backup formats.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3306

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-05-04 17:35:12 +00:00
neha_gupta
e4a587dc42
Update CHANGELOG.md for new release v0.7.0 (#3279)
<!-- PR description-->

Updating changelog for new release v0.7.0

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-02 10:21:49 +00:00
ashmrtn
c3cc40176e
Fallback to RepoRef is LocationRef is empty (#3270)
Fixes issue where doing backup detail or a restore
on an older backup and selecting by folder in
exchange wouldn't return any results. Root cause
is that older backup versions didn't populate
LocationRef in backup details which is what we
compare against

Manually tested on an older Corso backup. Will
write automated unit tests in a bit, wanted to
get this in prior to release

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3269

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-05-01 17:13:49 +00:00
ashmrtn
dcb9d81f3f
Fix progress folder display for Exchange (#3219)
---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3013

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-29 00:19:54 +00:00
Keepers
85f322fe97
don't give log-file prepopulate flag a default (#3237)
Passing the default logging file into flag pre-population causes the logger to assume the flag is always passed in and never falls back to the ENV setting.  This ensures that if no flag is provided, the env log file setting is used.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-27 00:54:15 +00:00
Keepers
f2f010b9f6
cache created folders when restoring drive items (#3220)
drive items aren't currently getting cached after creation.  Instead, we do a lookup on the folder name every time we walk the folder hierarchy toward the next child to restore.  Normally this would work fine, but apparently if the folder is named `folder`, then graph api returns nothing, causing the process to fail when repeatedly creating the folder.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-26 20:53:26 +00:00
Vaibhav Kamra
6395dcbe39
Add parent path information to malware detected items (#3221)
This allows identifying the file location when an item is detected with malware

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #3112 

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-26 17:17:52 +00:00
Keepers
7b1ce22a64
enable user pn->id migrations for reporef prefix (#3200)
Enables migrations of all user-based repoRef prefixes away from the user's PrincipalName and onto the user's ID for a more stable reference.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2825

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-24 22:37:11 +00:00
ashmrtn
9a8ec099cb
Handle subfolder moves and parent deletions right (#3186)
Properly merge items if a subfolder is moved,
the original parent is deleted and recreated,
and the subfolder is moved back to where it
started

See linked issue for a more detailed example

Manually tested original issue and fix on
OneDrive backup
---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3185

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-21 16:48:12 +00:00
Keepers
09ef350bc0
retry on "connection reset by peer" (#3138)
retry when we receive a "connection reset by peer" error

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3129

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-04-19 18:51:29 +00:00
ashmrtn
9664afaa12
Restore with OneDrive/SharePoint ID (#3151)
Allow restoring OneDrive and SharePoint items by
M365 ID as well as file name and ShortRef

Does create an edgecase where attempting to
restore an item named the same as another item's
M365 ID causes both items to be restored

Moves the consts for OneDrive file suffixes to a
different package so that the selectors package
can also import them without a cycle

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3142

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-19 15:30:03 +00:00
ashmrtn
1826b66e8d
Populate and persist OneDrive LocationRef (#3111)
Leverage structs from previous PRs to
populate and persist the LocationRef
for OneDrive

Right now the source of the
LocationRef is still based off the
RepoRef path, but this is in the
OneDrive code only

Manually tested
* updating a subfolder path when parent folder was moved
* populating LocationRef when the base backup didn't have LocationRefs when nothing was changed in OneDrive
* populating LocationRef when the base backup didn't have LocationRefs when something was changed in OneDrive

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2486

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-14 21:31:16 +00:00
Abin Simon
6c64eb0c16
Fetch individual attachments in case of full download failure (#3086)
Failure can happen as the list of attachment and they data can get
pretty big.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-04-13 05:51:58 +00:00
Abin Simon
c696e7b7ff
Enable permissions backup by default (#2981)
Previously we had a flag that needed to be flipped to backup permissions. Now we always backup permissions.

- Should we wait till https://github.com/alcionai/corso/issues/2976 is done to do this
- Users will have to do a non incremental backup first as of now to get permissions. We should probably have some way to force this.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/2409

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-04-04 03:55:44 +00:00
ashmrtn
16d1e68ebf
Populate location for all Exchange data types (#2829)
Always populate the location field of backup details
for Exchange data types

Fix bug where ParentPath for calendar items used
folder IDs instead of display names

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2486
* closes #2827

merge after:
* #2826
* #2808
* #2828

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-04-03 20:22:31 +00:00
Abin Simon
446ccfe491
Skip large OneNote items that cannot be downloaded (#2925)
OneNote files >2GB cannot be downloaded via the graph API. This change will gracefully skip them and prevent the backup from erroring out.

A OneNote file is represented by a Package(folder) which contains two file within it, `<file>.one` and `<file>.onetoc2`.  From what I can tell the `onetoc2` file is the metadata file and the `one` file is the one that contains data and can get big.

**The current implementation has a limitation(or feature depending on how you see it) in that we will end up doing a partial backup with just the `onetoc2` file but avoiding the big `one` file if that is big.**

I did not try to change the behaviour as even currently if for somehow we ended up with an error like 404 when trying to backup one of these files, we do skip it. I've added a comment in for now, let me know if we should go back and revisit the behaviour. *The main issue is the fact that we only hit this problem once it reaches kopia and removing it at that point is harder.*

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2910

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-31 04:43:34 +00:00
Abin Simon
43e16b6ab2
Fix flags for Exchange and OneDrive (#2978)
While we were reading flags from the cli, they were not being passed onto the restore path.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-29 22:36:48 +00:00
Keepers
8acff1ad93
fix sharepoint library folder selection (#2962)
Bugfix:
Sharepoint LibraryFolder scopes always return
zero matches in filtering with a prefix due to the
path not having the drive root prefix truncated, as is
expected of onedrive-based storage paths.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-03-28 01:21:20 +00:00
ashmrtn
ac2a299c51
Update to head of kopia for compression (#2953)
Pull in later version of kopia so we get the
ability to compress data from `StreamingFile`s
during upload. This doesn't compress data from
previously uploaded files

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2850

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-28 00:31:18 +00:00
ashmrtn
d1ce661b8f
Fixup item deletion edge cases (#2920)
Mostly centers around handling item deletion
when the item is both created and deleted
during item enumeration

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2917

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-22 23:31:46 +00:00
Abin Simon
7a7933e54c
Assume that item has no attachments if there is no body (#2911)
Some response from the Graph API was returning empty body. This could be something left over from older version of the platform, but this causes issues in our codebase.

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-22 22:53:45 +00:00
ashmrtn
d390aaae40
OneDrive invalid delta set difference (#2918)
When finding the set difference if the delta token has
expired do so by comparing folder IDs instead of paths.
This will keep from accidentally re-rooting subfolders
from a deleted folder onto a new folder at the same
path. This is safe because we're enumerating all files
and folders, so if the subtree wasn't actually deleted
it should be seen

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2915

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-22 22:19:58 +00:00
neha_gupta
adc4bb068a
continue if collection path is nil (#2912)
Currently code panics if it tries to find FullPath for a drive item and its not present.
Added a continue and skipped the item if FullPath is not present.

Root cause of bug is caused by having a folder in the previous paths map, having
the delta token expire, and seeing the folder be deleted in the delta results

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-22 21:11:21 +00:00
Abin Simon
e09a98f462
Fix permissions restore for version 6 (#2908)
Paths were not properly augmented for newer versions which removed the name from the `.dirmeta` file. This fixes that. It also adds a tests to ensure that the code will fail whenever we add a new version forcing us to revisit and update.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/pull/2906
* fixes https://github.com/alcionai/corso/issues/2909

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-22 20:27:17 +00:00
Abin Simon
8bebf579a4
Remove global 10 day timeout (#2875)
This has been fixed upstream with https://github.com/microsoft/kiota-http-go/pull/71 . We don't need this hack anymore.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/2849
* https://github.com/alcionai/corso/pull/2694
* closes https://github.com/alcionai/corso/pull/2893
* closes https://github.com/alcionai/corso/pull/2892
* https://github.com/alcionai/corso/pull/2891

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-22 13:37:30 +00:00
Vaibhav Kamra
f5bdbc7be2
Update CHANGELOG.md (#2905)
Update CHANGELOG - remove v0.6.0 and update with v0.6.1

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-22 04:00:47 +00:00
neha_gupta
7fbe6d1099
Update CHANGELOG.md (#2894)
<!-- PR description-->

update changelog for latest release

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-21 08:01:44 +00:00
Abin Simon
ef5178668f
Log user name when enumerating drive items (#2872)
This mimics the behavior for exchange where we print out which user and what content is currently being enumerated. While this might not be useful when we are only backing up a single user, it can come in handy when we run backups for multiple users.

<!-- Insert PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2871

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-21 03:51:16 +00:00
ashmrtn
a798932e4d
Add hidden flag to set fetch parallelism for Exchange from CLI (#2879)
#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2877

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-20 21:48:21 +00:00
ashmrtn
a13479a0f6
Enable OneDrive incrementals, disable incrementals for permission backup (#2642)
Permissions backup requires more work, so we only support delta incrementals for OneDrive if that's not enabled.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2333

Pending merge of
* #2650
* #2496

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-03-18 17:37:44 +00:00
Keepers
c28d5399ef
mark event instance loss as known (#2837)
#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🗺️ Documentation

#### Issue(s)

* #2835
2023-03-18 17:04:15 +00:00
Keepers
621e850531
add 10 day deadline to cli context (#2852)
Ensures all CLI usage has a context deadline so that we
don't bump into the graph api 100 second deadline.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
2023-03-17 23:48:13 +00:00
Abin Simon
0e1064078a
Backup owner and empty permissions (#2833)
Backup owner and empty permissions, we cannot restore them though (refer https://github.com/alcionai/corso/issues/2789#issuecomment-1471787829) . Previous we were just skipping even backing them up. It might be good idea to have this data in case we find that we are able to restore them in the future.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes https://github.com/alcionai/corso/issues/2789

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-17 09:42:23 +00:00
Keepers
8dfb00f308
add fault errors streamstore (#2731)
Adds a new streamstore controller for fault.Errors. This provides large scale, extensible file storage for fault errors to be persisted, much like we do for backup details.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2708

#### Test Plan

- [x] 💚 E2E
2023-03-17 01:07:07 +00:00
Abin Simon
0125876192
Skip items returning 404 (#2805)
Treat a 404 as an unavailable item and skip it similar to malware items.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
Treat a 404 as an unavailable item and skip it similar to malware items.
2023-03-15 17:52:22 +00:00
Abin Simon
efee7c4419
Fix item re-download on unauthorized requests (#2804)
We had a regression which caused item re-downloads on 401s from
initial delta URL to be silently skipped. This fixes that behavior.<!-- Insert PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-03-15 07:45:20 +00:00
Keepers
d9e9e89cca
Make sharepoint cli public (#2787)
Unhide sharepoint CLI commands, and do not
mark them as pre-release.  Hides the --data flag
since we only currently support libraries.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2757

#### Test Plan

- [x]  Unit test
2023-03-14 05:59:24 +00:00
Abin Simon
921542c0f5
Fix repo connect error when we don't have a config (#2771)
Previously we were failing to connect to the repo if we did not have an existing config file present but could not run init to generate it (had already initialized the repo).

Error: retrieving storage provider details: bucket is required to perform this command

<!-- Insert PR description-->

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Regression from https://github.com/alcionai/corso/pull/2534

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-03-14 04:34:34 +00:00
Abin Simon
81a37608a2
Post release changelog updates (#2767)
Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
2023-03-13 17:42:30 +00:00
Keepers
871f0b406d
introduce fault items (#2710)
Adds the item struct to the fault package for tracking serializable and dedupliatable error sources.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🌻 Feature

#### Issue(s)

* #2708

#### Test Plan

- [x]  Unit test
2023-03-10 17:23:19 +00:00
Keepers
07b4900f58
remove stats.Errs (#2591)
## Description

Now that fault errors is in place, we don't need
the operation stats errs.

## Does this PR need a docs update or release note?

- [x]  No 

## Type of change

- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1970

## Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-03-08 20:19:19 +00:00
Abin Simon
33ecbd1820
Switch to using ids for storing permissions (#2674)
This also enable backing up and restoring permissions for groups and applications.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/2621
* fixes https://github.com/alcionai/corso/issues/2673

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-03-07 04:07:37 +00:00
Keepers
d1404626f1
avoid downloading known malware (#2702)
If graph flags an item as malware,
avoid attempting to download it.
Only applies to drive items.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #2701

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
2023-03-06 20:28:10 +00:00
ashmrtn
aef90f8b86
Remove GraphConnector user population (#2582)
## Description

Don't populate users when GraphConnector instance is created

## Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No 

## Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

## Issue(s)

* #1994

## Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-02-28 21:26:12 +00:00
ashmrtn
04516692f4
Don't NPE if user isn't in tenant (#2678)
Was causing panics when trying to access kopia stats. Panic was recovered from, but reading the error was difficult.

Add some CLI tests to hopefully stop future regressions for this specific case

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #2668

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-02-28 20:13:34 +00:00
Keepers
1945db6b65
force omission of meta files (#2662)
While the `isMeta` flag will solve this in the
future, the current state requires us to filter
out all meta and dirmeta files, even historically.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #1970

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-02-27 22:46:55 +00:00
Keepers
cb12329d0c
don't handle meta files in details (#2606)
## Description

Remove the display and interaction with .meta
files from details entries.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🐛 Bugfix

## Test Plan

- [x]  Unit test
2023-02-23 04:14:17 +00:00
Keepers
129b96714b
fix url logging (#2617)
## Description

The `GetUri()` call contains side-effects which drain the builder of its constructed url data.  As a result, running a backup with CORSO_URL_LOGGING actually forces the bug to manifest unintentionally.

## Does this PR need a docs update or release note?

- [x]  Yes, it's included

## Type of change

- [x] 🐛 Bugfix

## Test Plan

- [x] 💪 Manual
2023-02-22 22:29:55 +00:00