No logic changes, just removing a toggle that's already been set.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2988
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3893
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Introduces a counting bus, and threads it into restore operations so that we can count the number of
collision skips that occur.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
implements cli integration for configuring restore operation collision policy and restore destination.
flags are hidden and will be made visible at a later time.
---
#### Does this PR need a docs update or release note?
- [x] 🕐 Yes, but in a later PR
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
initializes the concurrency limiter as part of CLI pre-run initialization to ensure we don't accidentally skip out on including the limiter in middleware.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🐛 Bugfix
#### Issue(s)
* #3695
#### Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
Use default values from endpoint only if its not present in config file
#### Does this PR need a docs update or release note?
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
<!-- PR description-->
Flags for all configs-
Azure cred flags- (azure-tenant-id, azure-client-id, azure-client-secret) present in -
- Backup (create, delete, details and list) and restore of Exchange, Onedrive and Sharepoint command
- S3 repo init and connect command
AWS cred flags - (aws-access-key, aws-secret-access-key, aws-session-token) present in-
- Backup (create, delete, details and list) and restore of Exchange, Onedrive and Sharepoint command
- S3 repo init and connect command
Passphrase flag- (--passphrase) present in-
- Backup (create, delete, details and list) and restore of Exchange, Onedrive and Sharepoint command
- S3 repo init and connect command
S3 flags-
--endpoint, --prefix, --bucket, --disable-tls, --disable-tls-verification - flags is for repo init and connect commands
all the S3 env var will also work only in case of repo init and connect command. For all other commands user first connects to repo. Which will store the config values in config file. And then user can use that config file for other commands.
No cred configs are save in the config file by Corso.
Config file values added-
Azure cred -
- azure_client_id
- azure_secret
- azure_tenantid
AWS cred -
- aws_access_key_id
- aws_secret_access_key
- aws_session_token
Passphrase -
- passphrase
**NOTE:**
- in case of AWS creds all the three values should be provided from same method. Either put all values in env, config file and so on.
- all the S3 env var will also work only in case of repo init and connect command. For all other commands user first connects to repo. Which will store the config values in config file. And then user can use that config file for other commands.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3522
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
The goal of this PR is to normalize the cli packages in a way that 1/ showcases clear ownership of data, 2/ minimizes package bloat, and 3/ helps avoid circular import issues.
To achieve this, two primary changes were made.
First, the cli/options package was folded into cli/utils, so that all "shared functionality" is owned by a single package. Second, all flag values, globals, declarations, and mutator funcs (in the cli layer, logging package was not changed) were extracted from cli/utils and placed into cli/flags. This divides ownership between the declaration and population of the flags (cli/flags) from the utilization of values derived from flags in command processing (cli/utils).
This PR contains zero logical changes. Only code
movement and renaming.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3664
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
Some events in have a empty repoID. This could be because of multiple
config files used. And new config files might have repoID missing.
Solution- Fetch repoID from Kopia is not found locally in the config
file.
Note: Since the Corso Start event happens before we connect to Kopia, it
might have repoID missing if the config file is used for the first time.
All the consecutive events will have correct values. Corso start event
will start populating correct values if the config file is used once.
#### Does this PR need a docs update or release note?
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🐛 Bugfix
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/3388
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
centralizes all restore configuration management within a restoreConfig struct. This struct is owned by the control package, which allows it to be utilized by both CLI and SDK consumers.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3562
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
For large restore, it just ends up printing a lot of text. This change makes it print a summary instead. We still retain printing the full content if the user asks for a json output as we assume they would be piping it somewhere.
```
Connecting to repository: 1s done
Restoring to folder Corso_Restore_24-May-2023_11-49-44
Connecting to M365: done
Restoring ∙ 7ceb8e03-bdc5-4509-a136-457526165ec0
Discovered 3 items in backup d5c07808-545d-4906-9ade-f224ca8f13de to restore
Enumerating items in repository: 0s done
Restoring data: 54s done
Restored 304 items.
```
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
updates and improves cli docs. in part for easier reading and explanation. in part to remove all references to resource owner flags in details and restore.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
#### Type of change
- [x] 🐛 Bugfix
#### Test Plan
- [x] 💪 Manual
moves common/time.go to common/dttm/dttm.go, and renames many of the consts and funcs with names that better explain their design and intent of usage.
Only code movement, no logical changes.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Move code for connecting to a repo into a common
package so that backup and restore CLI code can
both use it
This will also make it easier for maintenance
code in the future as it can reuse the same
helper
There are no logic changes in this PR, only
code movement
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
tangentially related to
* #3077
#### Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Cleans up and adds unit tests for options flags.
Removes variadic func builder pattern for single
func versions.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #2024
#### Test Plan
- [x] ⚡ Unit test
Appends FV to the shared values populated by
cobra flags. This helps ensure those values are
not confused by other, like-named values used as
func params or local vars. Also amends some
other varname issues.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Test Plan
- [x] ⚡ Unit test
While we were reading flags from the cli, they were not being passed onto the restore path.
---
#### Does this PR need a docs update or release note?
- [x] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.
Might follow up with a linter to enforce this change.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
Updates the sharepoint cli examples to better
reflect the current support and utilities.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🌻 Feature
#### Issue(s)
* #2786
#### Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Github workflow - Release sanity test which can run pre and post
release.
Testing-
- version command
- repo init and connect
- backup and restore of exchange and onedrive
- verify correct no. of emails restored
- verify file sizes restored and roles on folder are correct.
- incremental backup and verification of the same
**Further enhancement-**
- check actual email data instead of just no. of emails
- verify contacts and calendar
- permission in onedrive
- slack notification on failure
- if we recieve an email in between a backup and restore, we will fail
the test as the counts won't match
**issues for handling enhancement-**
https://github.com/alcionai/corso/issues/2743https://github.com/alcionai/corso/issues/2742
## Does this PR need a docs update or release note?
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🤖 Test
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic
words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/pull/2622/
## Test Plan
- [ ] 💪 Manual
## Description
Trigger an event at start of every CLI command.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
## Issue(s)
* https://github.com/alcionai/corso/issues/1761
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
---------
Co-authored-by: Keepers <ryanfkeepers@gmail.com>
Co-authored-by: aviator-app[bot] <48659329+aviator-app[bot]@users.noreply.github.com>
## Description
Create a wrapper struct that provides a `Fetch(ctx, name) (Stream, error)` function that always returns `ErrNotFound`. A future PR is going to expand the `RestoreCollection` interface to include that function and I wanted to reduce the amount of chaff that would come out of it
This PR just creates the wrapper and moves `ErrNotFound` from the kopia package to `data` package to avoid import cycles
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [x] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1944
## Test Plan
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Removes references to M365 ID as instruction.
<!-- Insert PR description-->
## Does this PR need a docs update or release note?
- [x] ⛔ No, self-documenting
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🐛 Bugfix
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* closes#2190
## Test Plan
- [x] ⚡ Unit test
## Description
Ensures that the details and restore commands
exit without attempting further processing if a
backupID doesn't point to either a valid backup
or backup details, and that the user is appropriately notified.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🧹 Tech Debt/Cleanup
## Issue(s)
* #1877
## Test Plan
- [x] 💚 E2E
## Description
Now that resource owners are identified via
the selector itself, rather than each scope, we
can remove the resource owner data from
scope production and data.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Checks for resource owner matches in the top
of the reduce func using the selector owners,
instead of waiting until the path match check.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
selector creation now includes a parameter for
a slice of resource owners (users or sites). This
is step one in migrating resource owner lists out
of scopes and into the selector. next step is to
have the selector utilize the primary list instead
of the per-scope list.
## Does this PR need a docs update or release note?
- [x] ⛔ No
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1617
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Adds restore commands to the cli for sharepoint.
The restore process is only partially functional at this time.
Library files that pass auth are able to be restored as expected.
However, auth issues (not directly related to these changes) prevent
restoration of all library items at this time.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #1615
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
Remove old "restored..." confirmation line from the CLI. Renamed the "Reference" column header to "ID" in the tabular output, which represents the shortRef aka item id.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1269
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
Adds a processor that confirms whether user has
added a value for a flag in the cmd, or if it is the
default value. This map of valued flags is added
to the service opts structs to for validation.
Also migrates many service flags to utils as
consts so that these values can be maintained
as consistent across packages.
## Description
Wrong service referenced in restore output
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* closes#1099
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds validation of cli inputs for time-based flags.
All time values that can be parsed by the time
handling in common count as supported, even
though official support includes a smaller set.
Also adds some clean-up to time.go and adds a
design document describing standard time value
formats, and the maintenance thereof, in corso.
## Type of change
- [x] 🌻 Feature
- [x] 🗺️ Documentation
- [x] 🤖 Test
## Issue(s)
* #943
* #924
## Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Removes the -all flag from the exchange cli. The
same functionality can be gained with the flag
--user *.
Additionally does some minor tidying of other
comments and outputs throughout the cli.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #1032
## Test Plan
- [x] ⚡ Unit test
## Description
Previous indent fixed usage but screwed up generated docs.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [x] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Adds Examples to CLI usage output and the auto-gen docs
Also cleans up:
* OneDrive flags and flags help
* Addition of OneDrive generated files to docs
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Fixes#529
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Now that GC is tracking the details entries during
a restore procedure, it can display the results in
the cli. Due to an absence of itemInfo, only the
item shortRef is displayed. But we can expand
on that from here.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #977
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
* Clean up pass on all CLI output
* Added basic extended usage
* Mark required params in generated docs
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes#527
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Disable automatic flag sorting and explicitly order based on something more sensible. Fixes the order in both inline help output and docs.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* Fixes # 783
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
* No new functionality exposed to CLI users
* generate restore folder names in CLI and pass down the stack
* update tests for new parameters
* centralize generation of restore container names
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
* #897
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
Consolidate function parameters by passing them as a single struct instead of a bunch of parameters. Helps keep function signatures shorter and also makes it easier to modify in the future (just change the struct rather than all call sites).
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🐹 Trivial/Minor
## Issue(s)
* #913
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
removes the requriement that higher order data
must be specified in order to specify granular
data ids. Ex: previously, cli users needed to add
--user and --folder flags in order to use a --mail
flag. Now the user and folder will default to Any()
if they're not provided.
## Description
Adds foundational cli integration tests for backup/restore
of events and calendars
## Type of change
- [x] 🤖 Test
## Issue(s)
#501
## Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.
All other changes in the repository update the
package imports to the new module path.
## Description
Adds the new event filter selectors to the CLI for
backup detail and restore commands.
## Type of change
Please check the type of change your PR introduces:
- [x] 🌻 Feature
## Issue(s)
#501
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
