<!-- PR description-->
New commands to initialize & connect to a repo on local or network attached storage.
* `repo init filesystem --path /tmp/repo`
* `repo connect filesystem --path /tmp/repo`
Includes basic unit & e2e tests. More coverage to be added in a following PR to keep the size contained.
**Updates:**
* Added Repo path sanitization i.e. handle relative paths, make paths cross platform compatible, etc.
* Removed retention artifacts, not supported for filesystem storage.
* cli docs - auto updated.
* Manually tested with all corso backup/restore/export commands.
**Doesn't include**
1. Symlinks
2. User ids wiring into repo.
3. Repos documentation update - in an upcoming PR.
4. Prefix support -> kopia doesn't support prefixes for `filesystem` storage
5. More E2E tests.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1416
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
Introducing a new `Configurer` interface to abstract out storage config information(for s3, filesystem etc) from caller code. I consider this as a short term solution. We need to consolidate overall config handling in a better way, but that's out of scope for this PR chain.
Testing
* Most of the changes here are code movement under the hood. So relying on existing tests.
* I'll address any test gaps in a later PR.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1416
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
One of the 2 remaining setup PRs before we can introduce local storage repos.
**Changes:**
1. Read storage provider(`provider`) from config file except for `repo init * ` or `repo connect *` commands.
2. Apply flag overrides based on provider type( e.g. `S3FlagOverrides` if provider is `S3`)
3. Propagate storage provider type to functions which read/write config. These functions arbitrate on config hierarchy - flags, env, config file, in that order.
**Reasons**
* Reason 1 is needed is because config file is the source of truth for storage provider for all commands except `repo init` or `repo connect`. In the exception cases, we pick the provider in command (e.g. `s3`) as the source of truth. e.g. consider a `repo init s3`, followed by `repo init filesystem`.During `repo init filesystem`, config file would indicate `S3` provider, but the correct behavior here is to select `filesystem` provider.
* One alternative was to push provider from the init/connect cmds into an override flag, and let the config code decide on hierarchy. However, this felt hacky. provider here is not a flag to begin with. It's part of init/connect commands.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1416
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
<!-- PR description-->
One of the setup PRs before introducing local storage support for corso.
Changes
1. Export `storage.storageProvider` and rename it to `storage.Provider`. Without the rename, linter errors out with `exported: type name will be used as storage.StorageProvider by other packages, and that stutters; consider calling this Provider`
2. Rerun stringer - `stringer -type=Provider -linecomment`
3. Introduce a new Provider - `FS` for filesystem. This is in line with kopia naming for local/network attached storage.
4. Fix hardcoded string values for storage providers in test code.
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
I've needed to catch gotchas that arise from contributors adding a value in the middle of an iota list, not to mention have dealt with prior bugs that happened the same way, now too many times to feel safe about its usage.
This PR removes the use of iota from all const declarations. The intent is to not allow the use of iota within the codebase.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #3993
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
<!-- PR description-->
S3 configs like BUCKET and PREFIX are common env vars and can cause overkills. Remove these S3 configs to avoid confusion.
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
Allows caller to specify a IAM role to assume in the Kopia storage provider
---
#### Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
#### Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Supportability/Tests
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
#### Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2106
#### Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
Mostly find/replace on errors.N and errors.W. Also turns all wrapf into wrap, and removes as many errorf calls as possible.
Might follow up with a linter to enforce this change.
---
#### Does this PR need a docs update or release note?
- [x] ⛔ No
#### Type of change
- [x] 🧹 Tech Debt/Cleanup
#### Issue(s)
* #1970
#### Test Plan
- [x] ⚡ Unit test
- [x] 💚 E2E
## Description
This PR removes the option to specify a credential provider since that option didn't pan out.
Also reverts back to using kopia upstream (switch to HEAD from `main` in prep for an upcoming fix)
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [ ] 🕐 Yes, but in a later PR
- [x] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [x] 🧹 Tech Debt/Cleanup
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
Implements support for a caller to specify a credential provider.
This assumes an S3 credential provider but we should follow-up with a change that isn't S3 specific.
## Does this PR need a docs update or release note?
- [ ] ✅ Yes, it's included
- [x] 🕐 Yes, but in a later PR
- [ ] ⛔ No
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🧹 Tech Debt/Cleanup
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #2106
*
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
## Description
Introduces config options (`--disable-tls` and `--disable-tls-verification`) to turn off TLS and TLS verification
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [x] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1415
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [x] ⚡ Unit test
- [ ] 💚 E2E
swaps the corso go module from github.com/
alcionai/corso to github.com/alcionai/corso/src
to align with the location of the go.mod and
go.sum files inside the repo.
All other changes in the repository update the
package imports to the new module path.
* Fix wsl lint errors in pkg package
* Fix wsl lint errors in most of internal package
Leave some sub-packages out that have higher churn at the moment.
* Basic golangci-lint config with gofmt and timeout
Remove corresponding sections from other CI config files.
* Enable comment and string spelling linter
* Fix spelling mistakes for linter
Co-authored-by: Danny <danny@alcion.ai>
This allows the user to specify repo configuration via environment variables when a config file is not available.
This is a temporary fix - ideally we would leverage Viper for this but we currently only use Viper if a config file
is available so that needs to be refactored a bit.
[1]
`TENANT_ID` (already supported)
`BUCKET`
`ENDPOINT`
`PREFIX`
* move config unions to common code
The configuration union handlers in Storage and Account
overlapped significantly in behavior. Moving those helpers into
a common code folder was requested. Although the behavior
was similar across the files, the types were not, requiring
the addition of generics to solve the need.
* validate required storage props (#85)
Centralizes validation of required storage config properties within
the storage package. Requiremens are checked eagerly at
configuration creation, and lazily at config retrieval.
Additionally, updates /pkg/storage tests to use suites
and assert funcs.
* add validation failure tests to storage
* add common config and encryption passwd
Adds a provider-independent configuration handler, and the
the encryption password config property. The password is used
to encrypt and decrypt the kopia repository properties file.
* fix corso_password in ci.yml
* actually use the corso password in testing
* replace passwd in ci.yml with a secret
* ci.yml secret typo fix
* introduces storage pkg (#25)
Adds a new package /pkg/storage. Storage is used to
communicate storage provider information throughout the corso
app. In particular, it allows per-provider data to cross interface
boundaries without slicing their details.
* use consts for s3 config keys
