## Description
Three primary changes:
1. move the list of load test users into a secret in the github test domain.
2. purge all data that was generated for users in the load test
3. fix all `m356` typos.
## Type of change
- [x] 🤖 Test
- [x] 💻 CI/Deployment
## Issue(s)
* #1266
## Test Plan
- [x] 💪 Manual
- [x] 💚 E2E
## Description
We need cli docs to be available to be able to be able to build docs.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/pull/1434
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
We can probably remove the requirement on `Publish-Docs` as we have move the check to `Docs-Linting`. Having this check in `Publish-Docs` does not run for docs only change as the previous steps(Test-Suite) are skipped and as a result this gets skipped.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Version reported for binaries inside docker images was incorrect. This fixes it and sets it to follow https://github.com/alcionai/corso/issues/29
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Push docs to root instead of /preview
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* fixes https://github.com/alcionai/corso/issues/1327
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Pushes `nightly` as the tag for latest images.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
There was a missed link to tutorial in main page in https://github.com/alcionai/corso/pull/1245. Fixes that. Also enabled docs build (not push) for docs in CI as that is pretty fast anyways. Fixes `Push-Docs` job failing in `main`.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [ ] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
Followup for https://github.com/alcionai/corso/pull/1240
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [x] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/pull/1240
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
While trying to work on https://github.com/alcionai/corso/issues/1166, I ended up reworking parts of the build script ~, but I'm running into some issues with building it on CI. Pushing it here just for reference.~
The new one combines both `build.sh` and `build-container.sh` into a single script where you can specify what to build. Also, inorder setup a proper multi arch, build system locally, we will have to properly setup buildx which is explained in https://docs.docker.com/build/building/multi-platform/ or https://stackoverflow.com/a/70837025/2724649 . I'll add this instructions to docs.
The new build script looks something like this:
```
Usage: build.sh <binary|image> [--platforms ...] [--tag ...]
OPTIONS
-p|--platforms Platforms to build for (default: linux/amd64)
Specify multiple platforms using ',' (eg: linux/amd64,darwin/arm)
-t|--tag Tag for container image (default: alcionai/corso)
```
---
I've made sure the image and binary has the proper architecure and that the amd64 one runs properly in my system. It would be helpful if someone who has access to arm system can validate the arm image. You can use https://github.com/alcionai/corso/pkgs/container/corso/45878348?tag=84fc9d4 image to verify.
```
$ cat check-image.sh
imgid="$(docker create "$1")"
docker cp "$imgid:corso" /tmp/corso
echo Image: "$(docker inspect "$1" | jq '.[0].Architecture')"
echo Binary: "$(file /tmp/corso)"
$ ./check-image.sh ghcr.io/alcionai/corso:84fc9d4@sha256:2278a2b4f108e5dd2ae545f53da1d151b77171f969ffd9718e4bb9886e332ee2
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
Image: "arm64"
Binary: /tmp/corso: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, Go BuildID=7iTBXW0reyfIOS-b-ciS/a5K0Q1IjuA0m9DJxmuNk/Ju1lI6bUZeKn6M_xqon6/KNXwYSnL7e5RVtjAKW9A, not stripped
$ ./check-image.sh ghcr.io/alcionai/corso:84fc9d4@sha256:6320b95470014ca07b9cf1db98b73f5672870c2c53c22c3d13223d88fa621ee0
Image: "amd64"
Binary: /tmp/corso: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, Go BuildID=sBziMHPFI9K-G0et0WjJ/Pe9A2Vy8_xpV3FEDJUMo/p4UeMEzgheASvylZ1N3j/fnwmDeVif4rhneou-S6O, not stripped
$ docker run -it --rm ghcr.io/alcionai/corso:84fc9d4@sha256:6320b95470014ca07b9cf1db98b73f5672870c2c53c22c3d13223d88fa621ee0
[...help message...]
```
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1166
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
migrates the client_id, client_secret, and tenant_id
environment variables to versions prepended with
azure_*.
⚠️ Your local environment will need to change in the following ways: ⚠️
1. prepend `AZURE_` to your clientID, clientSecret, and tenantID env vars.
2. update those same env references in any `.corso.env` files used for copying envs to docker containers.
3. in `.corso.toml` (and any variants such as .corso_test.toml), replace `tenantid` with `azure_tenantid`.
## Type of change
- [x] 🐹 Trivial/Minor
## Issue(s)
* #558
## Test Plan
- [x] 💪 Manual
- [x] ⚡ Unit test
- [x] 💚 E2E
* Make corso respect --version
* Add proper version information when building
* Use unified version string
$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)
With tag: v0.0.0-ceaf04c9
Without tag: unreleased-ceaf04c9
* Add a simple build command to build using Makefile
## Description
This sets up the metrics configs using `ldflags`. Keeping it in draft as I wanted to wait till https://github.com/alcionai/corso/pull/1052 is merged as that will affect how we use it in the CI. I have currently set the base branch to `release-ci` on GH, the branch for https://github.com/alcionai/corso/pull/1052 instead of `main` as the diff would make more sense that way.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #1067
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
This makes it so that we have an option to locally run all Makefile targets without having to go through Docker. That said, it retains Docker as the default way to run it, but now we can set an evn variable `CORSO_USE_DOCKER=-1` to skip running through docker. I understand if this looks a bit hacky and don't want to add it in, but thought I wold propose this anyways. While not major, I was able to get a good amount of decrease in build times.
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
If any of the dependents were skipped, the CI would skip a job. This resulted in `Docs-Linting` getting skipped because of `Generate-CLI-Docs` job being skipped for doc only change. Now they are both merged into a single job. The internal steps for cli docs generation will still be skipped, but that should not affect the steps below.
*I have not been able to fully test it out as the "file change checker" logic can only be checked after merging it, but it should be working*
Ref: https://github.com/orgs/community/discussions/25224
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
* Setup CI for release
* Initial commit of CHANGELOG.md
* Pin vale and markdownlint-cli to current versions
* Update branch name for CI
* Pull request workflow job
## Description
load testing for the dev m356 tenant takes longer
than 3 hours to complete, causing all load tests to
get canceled when github shifts priority to the
next job that gets kicked off. Backing off the freq to
every 4 hours, and will revisit that as needed. If 4 hour
time doesn't satisfy the boundary, we should shift the
dataset instead of freqnency.
## Type of change
- [x] 🤖 Test
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
add diagnostic records to load test runs
## Type of change
- [x] 🤖 Test
- [x] 💻 CI/Deployment
## Issue(s)
* #902
## Test Plan
- [x] 💚 E2E
## Description
Github cache imposes a 10gb limit, after which cache entries
will automatically get evictions (LRU policy). Various recent
queries have pinned our current cache size at 14-19 gb. The
golangci-lint cache alone accounts for 5-9 gb of that usage.
Meanwhile, it takes a shorter time for that package to download
missing modules than it does to run our tests. This change shuts
down the golangci cache in hopes that it improves our branch
caching retention.
## Type of change
- [x] 💻 CI/Deployment
## Issue(s)
* #790
## Test Plan
- [x] 💚 E2E
## Description
<!-- Insert PR description-->
## Type of change
<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor
## Issue(s)
<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>
## Test Plan
<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
An attempt to minimize and centralize how we're caching go builds.
Not necessarily going to solve all our CI issues, but should make
changes building on this state much easier.
## Type of change
- [x] 💻 CI/Deployment
## Issue(s)
* #790
## Test Plan
- [ ] 💪 Manual
- [ ] ⚡ Unit test
- [x] 💚 E2E
## Description
The current action workflow skips cause requried
actions to be marked as passing until they fail,
not marked as incomplete until either passed,
failed, or skipped. This change should provide
the latter behavior.
## Type of change
- [x] 🌻 Feature
## Issue(s)
* #744
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
## Description
Re-opening the PR with a hack (sourced from github
themselves) to skip the required check if we don't run
the standard required jobs.
## Type of change
- [x] 🌻 Feature
## Issue(s)
#744
## Test Plan
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
* Basic golangci-lint config with gofmt and timeout
Remove corresponding sections from other CI config files.
* Enable comment and string spelling linter
* Fix spelling mistakes for linter
Co-authored-by: Danny <danny@alcion.ai>
* adds env-based integration test controls
Some integration tests were running locally without being told
to do so via env variables. This adds some controllers for validating
if and when those tests should run, and for ensuring the env
variables necessary to run the test are available.
* testing helper comments, func name change
* add testing timeOf log, skip gc ci tests
* corrected the test skip in GC_test
* update kopia_test with helpers
* rename /testing files to *_test.go
This will ensure these files do not compile into the release binary.
* Revert "rename /testing files to *_test.go"
This reverts commit 04fd2046cc6d11bf3e3eb98d98a3e84b1bba6366.
* add common config and encryption passwd
Adds a provider-independent configuration handler, and the
the encryption password config property. The password is used
to encrypt and decrypt the kopia repository properties file.
* fix corso_password in ci.yml
* actually use the corso password in testing
* replace passwd in ci.yml with a secret
* ci.yml secret typo fix
