1398 Commits

Author SHA1 Message Date
ashmrtn
ab344422d6
Cleanup figuring out kopia retention parameters (#3857)
Create a struct that handles:
* initialization from existing kopia config info
* in-memory updates to config info
* detecting which config info structs from kopia need updated
* returning kopia config info structs

Overall, this allows us to isolate the logic for
calculating the new retention configuration info
in kopia

Viewing by commit may help.
First commit just splits up existing code, moving it into
either conn.go (will be used later) or retention/opts.go.
Subsequent commits switch to using a struct, add tests,
and fixup existing logic

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-07-24 18:43:27 +00:00
ashmrtn
c073ccc5ea
Add a couple more linters (#3886)
Lint for kv-pairs in log calls and things
that can be replaced with constants from
the standard library

Also add the linters that are enabled by
default just so things are more consistent
and clear going forward

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [x] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3654

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-07-22 01:34:38 +00:00
Keepers
0d6b08204d
allow users to limit page size (#3875)
allows cli users to limit the page size of delta queries by calling a new hidden flag: --delta-page-size.
This also adds the control.Options struct to the api client, so that configurations such as this can be easily handed into, and used by, the client.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-07-22 00:00:09 +00:00
ashmrtn
62d4c68c04
Have kopia package generate tags during backup based on Reasons (#3869)
Move tag generation from the backup op to the
kopia package. This makes it match the pattern
that finding base backups uses where a set of
Reasons and a separate set of additional tags
are provided

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2360

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-07-21 22:49:42 +00:00
Keepers
7677299ace
use replace collisions for nightly tests (#3882)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-21 22:19:13 +00:00
ashmrtn
916bb0b27c
Create an interface and implementation for Reason struct (#3868)
Intermediate step to a few different goals including
* moving interface definitions to better locations while avoid cycles
* adding a flag to disable kopia-assisted incrementals

Create an interface and implementation for the existing Reason
struct. The goal is to set stuff up so that eventually the kopia
package can ask the struct for the subtree path to work with
when merging the hierarchy instead of having the backup operation
pass that information in

Code changes are mostly just turning stuff into a struct and
fixing up compile errors. Some functions have been excluded from
the struct (i.e. `Key`) and made into functions in the kopia
package itself

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #2360

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-07-21 21:05:38 +00:00
Keepers
d555501093
add integration tests for missing drives (#3847)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3652

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-21 01:13:09 +00:00
Keepers
9359679f99
add api funcs for creating documentLibs (#3793)
Adds api handlers for creating document libraries in sharepoint. This is the first step in allowing us to restore drives that were deleted between backup and restore.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-20 21:05:30 +00:00
ashmrtn
2c6417d2eb
Kopia wrapper code to set retention parameters (#3792)
Code at the kopia wrapper layer that allows configuring kopia
for retention if desired. Includes some tests that require
running against a bucket with object locking enabled

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3519

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-07-19 20:50:17 +00:00
Keepers
5a78f478a1
add operations test for adv rest (#3783)
ads operations level tests for advanced restore
configuration on all three services.
Code is largely boilerplate between each service, but with just enough quirks that full consolidation would require excess jumping through hoops.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3562

#### Test Plan

- [x] 💚 E2E
2023-07-17 03:56:08 +00:00
Keepers
09e5e9464a
add test for 503 resp body (#3812)
#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* closes #3811

#### Test Plan

- [x]  Unit test
2023-07-14 23:40:30 +00:00
Keepers
fd6dff3270
retry on http/2 stream errors (#3802)
http/2 stream error failures don't allow
us to leverage client middleware.  As a
result we're lacking a retry handler.  This
change adds retries for http_wrapper
in the explicit case of a http/2 stream
error.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-07-14 22:57:51 +00:00
Keepers
e60610193c
specify sharepoint restore libraries (#3801)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💚 E2E
2023-07-14 01:21:53 +00:00
Abhishek Pandey
19d56b1e29
Return original read closer if no extension factories are supplied (#3784)
<!-- PR description-->

As title

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-07-11 21:46:33 +00:00
ashmrtn
83c18e5276
Add tests that boomerang items (#3779)
Add a test that will fail if something is wrong with tracking items that are moved from a folder to a different folder and back to the original folder between incremental backups. This is important as it will help us ensure we don't accidentally turn on Exchange immutable IDs before they're fixed on the Graph side

Only tests Exchange mail and OneDrive/SharePoint as other Exchange data types don't support move operations via Graph API (it is possible to move events between calendars in the web UI though)

Manually tested to ensure it does actually fail if immutable IDs are enabled

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3755

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-07-11 00:18:19 +00:00
ashmrtn
87b8631037
Fixups for comparing details to expected set (#3778)
Collection of fixes to get tests passing again including:
* ensure expected sets are tracked for the lifetime of the struct
* add check that will fail if there's no expected deets to compare against
* minor fixups to path building when reading in an entire details file
* trim down data sent for patch operations to only the data that's changed
* update how expected items are gathered -- just pull it from graph directly
* update expected written items for OneDrive -- most likely due to matching both test folders (e.x. `corso_incr_test_1` and `corso_incr_test_1/corso_incr_test_1`)

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* closes #3777

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-07-10 23:46:13 +00:00
ashmrtn
97c10cbc68
Skip failing SharePoint pages tests (#3781)
The SharePoint pages code is neither available from the CLI nor maintained right now. When the code is maintained again the tests can be unskipped.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3780

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-07-10 21:29:27 +00:00
Abhishek Pandey
83e56ed396
Integrate extensions into drive item reader (#3750)
<!-- PR description-->

* Integrates corso extensions into onedrive/sharepoint library item backup flows.
* Also includes integration tests for OD/SP, unit tests for collections.
* Includes a small fix for `rw *backupStreamReader Close()`

Remaining things which will be covered in later PRs:
* extension tests with incremental backups
* Observability related changes for extensions

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* internal
#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-07-06 23:01:18 +00:00
Keepers
5ea194dc87
tweaks and fixes found while testing (#3735)
a variety of small updates that came
from manual testing of restore with various
collision and destination combinations.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3562

#### Test Plan

- [x] 💪 Manual
2023-07-06 22:24:26 +00:00
Keepers
39a93e9054
move default folder consts to api (#3734)
only code movement, no logic changes

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-06 20:25:06 +00:00
Keepers
2a150cc610
add counter for skip-collision counting (#3722)
Introduces a counting bus, and threads it into restore operations so that we can count the number of
collision skips that occur.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-06 19:22:02 +00:00
Keepers
8ba79709a6
split tester into separate files (#3762)
This is primarily an exercise in reducing the number of circular imports we get from adding the tester package to other packages.

No logic changes.  Purely movement/renaming.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-06 15:43:57 +00:00
Keepers
fe6ae28b68
distribute operations integration tests (#3706)
Moves operations integration backup tests into a
subpackage, and slices up the test folder into
multiple, service specific folders.

No logic changes, only code copy-pasting and
renaming.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💚 E2E
2023-07-05 22:19:04 +00:00
Keepers
8cc3069a6a
ensure od and exchange can restore in-place (#3703)
Adds changes (mostly to exchange) that allow in- place item restoration.  Onedrive basically already supported in-place restores, however we weren't attempting them.  Exchange needed full support of GetContainerByName across its categories.

Next steps: add CLI integration to simplify manual testing.  Then add automation testing to exercise all in-place restore conditions and configurations.

---

#### Does this PR need a docs update or release note?

- [x] 🕐 Yes, but in a later PR

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-05 21:44:55 +00:00
Abhishek Pandey
6fff3f7d1d
Refactor populateItems (#3732)
<!-- PR description-->

Benefits:
* Narrowed scope & better testability. 
* Preps it for follow up PRs which will inject per-item corso extension interfaces
* Not adding unit tests yet, as the args & outputs of this new function will change in a follow up PR.
---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-07-05 18:52:21 +00:00
neha_gupta
09f3196ee0
Onedrive restore- handle unknown path error (#3643)
<!-- PR description-->
 In case of Onedrive Restore give a proper message if the path provided do not exist or restore is done on empty backup.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3579

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
2023-07-04 11:37:16 +00:00
Keepers
662d626809
handle file-folder collision cases (#3688)
handle onedrive item collision cases where a file to be restored is in conflict with an existing folder. Skip and Copy will function normally.  For Replace behavior, we'll defer to Copy in this situation, so that the file gets restored without changing or deleting the folder.  If a folder creation attempts to replace an item, we do a similar action and make a renamed copy of the folder instead.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-04 02:34:53 +00:00
Keepers
cf495a6c8f
add authErr to users mailbox check (#3744)
license issues can produce an authenticationError when retrieving a user's inbox.  This catches that error and treats it as a "mailbox not availble" condition.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3743

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-03 19:31:35 +00:00
Keepers
56151a82eb
remove discovery pkg (#3677)
Currently unused, and functionality is largely duplicated by the services/m365 package.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-07-01 13:52:57 +00:00
Keepers
8b81728488
fix concurrency limiter init race (#3715)
initializes the concurrency limiter as part of CLI pre-run initialization to ensure we don't accidentally skip out on including the limiter in middleware.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3695

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-30 23:54:54 +00:00
Keepers
5949a3bcd7
add logging indicating full backups (#3721)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Issue(s)

* #3720
2023-06-30 23:02:01 +00:00
Keepers
b74f65e301
properly declare resourceNotFound for sites (#3728)
Missing sites return an ItemNotFound error code,
which we aren't currently handling when doing
queries for individual sites.  This change catches that error code and stacks the error with a
resourceOwnerNotFound sentinel.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x] 💚 E2E
2023-06-30 18:14:06 +00:00
Keepers
f2a4fd0ba3
od: add per-container collision caching and checks (#3640)
pushes collision checks in onedrive upward from the api layer (due to the bug with ConflictBehavior) and into the restore controller.  Collision resolution in drive follows the same pattern that we use in exchange.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-30 16:13:35 +00:00
Keepers
c218dd865b
move event instance funcs to new file (#3659)
No logic changes, just code movement.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🧹 Tech Debt/Cleanup

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-30 15:50:04 +00:00
Abin Simon
9d801efa03
Backup and restore link share information (#3655)
As of now we are only backing up and restoring actual permissions. This adds support for backing up and restoring link shares as well.

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/3605

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2023-06-30 08:30:16 +00:00
Abin Simon
8683fbc067
Temporarily disable sharepoint permissions tests (#3724)
They seem to be acting up in the Ci

<!-- PR description-->

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-06-30 05:14:48 +00:00
Keepers
43d4a4d5e8
remove top query param in paged queries (#3719)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [x] 💚 E2E
2023-06-30 04:49:14 +00:00
neha_gupta
8c661164ad
add flags for azure and aws (#3590)
<!-- PR description-->
Flags for all configs-

Azure cred flags- (azure-tenant-id, azure-client-id, azure-client-secret) present in -
- Backup (create, delete, details and list) and restore of Exchange, Onedrive and Sharepoint command
-  S3 repo init and connect command

AWS cred flags - (aws-access-key, aws-secret-access-key, aws-session-token) present in- 
- Backup (create, delete, details and list) and restore of Exchange, Onedrive and Sharepoint command
-  S3 repo init and connect command

Passphrase flag- (--passphrase) present in- 
- Backup (create, delete, details and list) and restore of Exchange, Onedrive and Sharepoint command
-  S3 repo init and connect command

S3 flags- 
--endpoint, --prefix, --bucket, --disable-tls, --disable-tls-verification - flags is for repo init and connect commands
all the S3 env var will also work only in case of repo init and connect command. For all other commands  user first connects to repo. Which will store the config values in config file. And then user can use that config file for other commands.

No cred configs are save in the config file by Corso. 

Config file values added- 
Azure cred - 
- azure_client_id 
- azure_secret 
- azure_tenantid

AWS cred -
- aws_access_key_id
- aws_secret_access_key
- aws_session_token

Passphrase -
- passphrase

**NOTE:** 
- in case of AWS creds all the three values should be provided from same method. Either put all values in env, config file and so on.
- all the S3 env var will also work only in case of repo init and connect command. For all other commands  user first connects to repo. Which will store the config values in config file. And then user can use that config file for other commands.



---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ]  No

#### Type of change

<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature


#### Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3522

#### Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-06-29 05:30:15 +00:00
Keepers
f1c0b2b42f
return mail content size as restore size (#3705)
#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-28 22:45:15 +00:00
ashmrtn
dcf02f6256
Minor refactoring of graph error checking code (#3686)
Minor code cleanup to error checking and calling code:
- Remove unused function
- Update error check for folder not found
- Remove duplicate check in SDK function

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* fixes #3684

#### Test Plan

- [ ] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-06-28 21:21:28 +00:00
Keepers
d9f6a9297f
fix bad user info conditional on spo license (#3699)
Primary change is fixing the spo license check in
GetUserInfo from a || to a && !.  Currently the check returns the spo license error instead of treating it as a drive access condition.

Additionally adds a root site check in the m365
isServiceEnabled call, instead of always returning true for sharepoint.  This should catch spo license issues on a per-site basis, if someone makes it that far.

Finally, adds extra testing (plus some test code
consolidation and refactoring) to ensure GetUserInfo is properly tested.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3671

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-28 18:03:17 +00:00
Keepers
df100b9b3b
use errors.As to extract odataError (#3690)
casting only the top level error to an odataError
will slice out useful data when the odataError is
wrapped, stacked, or otherwise nested.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🐛 Bugfix

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-27 23:56:27 +00:00
Keepers
c7ce9f4c0e
make tests less sus (#3689)
#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x]  Unit test
2023-06-27 22:16:13 +00:00
ashmrtn
ce3ecacbe2
Add lint rule to use any instead of interface{} (#3685)
#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3654

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-06-27 20:34:30 +00:00
Keepers
10c8fd9d12
adds test coverage for events in exchange incrementals (#3661)
We still have events excluded from the exchange
incrementals integration tests.  For future safety, these need to be part of the testing group.

However, there's a hitch.  Our primary test user is broken again, and cannot retrieve any calendars
except for the well-known set: calendars, birthdays, holidays.  We'll have to fix that before merging.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🤖 Supportability/Tests

#### Test Plan

- [x] 💚 E2E
2023-06-27 20:05:25 +00:00
Keepers
f68fe90793
minor refactoring before changes (#3638)
renaming structs so that they don't follow the interface naming conventions, updating and expanding the test suite setup for api.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-27 19:26:17 +00:00
ashmrtn
4f9793a824
Handle broader set of user not found error codes (#3678)
graph has been inconsistent about what error code is returned
when the user doesn't exist. Expand the set of things we
check so that hopefully we'll get them all

Always return an error if the user doesn't exist as determined
by checking for exchange folders during GetInfo calls

Also add a little more stack trace info to returned errors

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* fixes #3672

#### Test Plan

- [x] 💪 Manual
- [x]  Unit test
- [ ] 💚 E2E
2023-06-27 18:59:33 +00:00
Keepers
2132b3c789
properly handle collision replace behavior (#3636)
completes the item collision handling behavior in
exchange by turning replace handling into a post-delete process.

---

#### Does this PR need a docs update or release note?

- [x]  No

#### Type of change

- [x] 🌻 Feature

#### Issue(s)

* #3562

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-27 16:17:37 +00:00
ashmrtn
81a289d8bc
Allow opening S3 repo at a specific point-in-time (#3657)
Thread the PointInTime argument through corso to kopia so that S3 repos can be opened at a specific point if they're using versioning and object locking

This only allows SDK access to the feature at the moment and no changes are made to enable immutable backups. This is just low hanging fruit since it's wiring

CLI changes will need more thought as we ideally only want to enable this behavior for a subset of S3 commands (restore, details, list)

Manually tested opening a repo with some hacky CLI code to wire up flags

---

#### Does this PR need a docs update or release note?

- [ ]  Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x]  No

#### Type of change

- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup

#### Issue(s)

* #3519

#### Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2023-06-27 15:25:54 +00:00
Keepers
0451e933d5
handle no sharepoint license err (#3673)
adds handling for graph api responses
where the tenant does not have an active
sharepoint license.  For user's drives, this
will fall under the same "service not enabled"
behavior we use to determine if the drive can
get backed up.  For sites, this causes the
"get all sites" call to return a serviceNotEnabled error.

---

#### Does this PR need a docs update or release note?

- [x]  Yes, it's included

#### Type of change

- [x] 🐛 Bugfix

#### Issue(s)

* #3671

#### Test Plan

- [x]  Unit test
- [x] 💚 E2E
2023-06-27 02:26:47 +00:00