34 Commits

Author SHA1 Message Date
Abin Simon
4b94c4f012
Option to not use docker for Makefile in docs/website (#1087)
## Description

This makes it so that we have an option to locally run all Makefile targets without having to go through Docker. That said, it retains Docker as the default way to run it, but now we can set an evn variable `CORSO_USE_DOCKER=-1` to skip running through docker. I understand if this looks a bit hacky and don't want to add it in, but thought I wold propose this anyways. While not major, I was able to get a good amount of decrease in build times.

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-10-14 03:18:50 +00:00
Abin Simon
523386a720
Merge CLI docs genertion and lint (#1130)
## Description

If any of the dependents were skipped, the CI would skip a job. This resulted in `Docs-Linting` getting skipped because of `Generate-CLI-Docs` job being skipped for doc only change. Now they are both merged into a single job. The internal steps for cli docs generation will still be skipped, but that should not affect the steps below.

*I have not been able to fully test it out as the "file change checker" logic can only be checked after merging it, but it should be working*

Ref: https://github.com/orgs/community/discussions/25224

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-10-13 01:31:42 +00:00
Abin Simon
658ebec0da
Update gotestfmt package repo in CI (#1103)
The package was moved to gotesttools org with the original one being
deleted soon.
2022-10-11 16:38:57 +05:30
Abin Simon
62c704a786
Fix issue with docs lint (#1102)
* Fix docs lint

* Fix log upload path
2022-10-11 08:50:11 +05:30
Abin Simon
1793cc3757
Setup CI for release (#1052)
* Setup CI for release

* Initial commit of CHANGELOG.md

* Pin vale and markdownlint-cli to current versions

* Update branch name for CI

* Pull request workflow job
2022-10-10 11:00:55 +05:30
Keepers
fa7f505b33
slow load test frequency to every 4 hours (#1039)
## Description

load testing for the dev m356 tenant takes longer
than 3 hours to complete, causing all load tests to
get canceled when github shifts priority to the
next job that gets kicked off.  Backing off the freq to
every 4 hours, and will revisit that as needed. If 4 hour
time doesn't satisfy the boundary, we should shift the
dataset instead of freqnency.

## Type of change

- [x] 🤖 Test

## Issue(s)

* #902

## Test Plan

- [x] 💚 E2E
2022-10-04 20:11:55 +00:00
Keepers
af12fba0c7
attempt local action for cache control (#1003)
## Description

Attempting to fix cache control.

## Type of change

- [x] 💻 CI/Deployment

## Issue(s)

* #790

## Test Plan

- [x] 💪 Manual
2022-10-03 23:52:36 +00:00
Keepers
011e24583f
add profiling, tracing to load testing (#971)
## Description

add diagnostic records to load test runs

## Type of change

- [x] 🤖 Test
- [x] 💻 CI/Deployment

## Issue(s)

* #902

## Test Plan

- [x] 💚 E2E
2022-09-29 02:01:15 +00:00
Keepers
04a8d10f75
address golang ci modfile discovery (#967)
## Description

Github cache imposes a 10gb limit, after which cache entries
will automatically get evictions (LRU policy).  Various recent
queries have pinned our current cache size at 14-19 gb.  The
golangci-lint cache alone accounts for 5-9 gb of that usage.
Meanwhile, it takes a shorter time for that package to download
missing modules than it does to run our tests.  This change shuts
down the golangci cache in hopes that it improves our branch
caching retention.

## Type of change

- [x] 💻 CI/Deployment

## Issue(s)

* #790

## Test Plan

- [x] 💚 E2E
2022-09-29 00:32:22 +00:00
Keepers
f10071189a
hide successful test outputs in ci action (#965)
## Type of change

- [x] 💻 CI/Deployment
2022-09-28 19:56:20 +00:00
ashmrtn
f27c3e0b83
Don't hide tests output, update archiver (#950)
## Description

<!-- Insert PR description-->

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* #<issue>

## Test Plan

<!-- How will this be tested prior to merging.-->
- [ ] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-27 13:37:25 +00:00
Keepers
a368570e20
improve ci cache management - initial cleanup (#849)
## Description

An attempt to minimize and centralize how we're caching go builds.
Not necessarily going to solve all our CI issues, but should make
changes building on this state much easier.

## Type of change

- [x] 💻 CI/Deployment

## Issue(s)

* #790

## Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-09-15 16:25:05 +00:00
Keepers
0c8a03a826
add the metrics package (#839)
## Type of change

- [x] 🌻 Feature

## Issue(s)

* #741

## Test Plan

- [ ] 💪 Manual
- [ ]  Unit test
- [x] 💚 E2E
2022-09-15 00:28:02 +00:00
Keepers
1295750206
use main version of fliechange checker (#858) 2022-09-14 18:39:24 -04:00
Keepers
b739526bb5
use dorny version of skipping workflow jobs (#768)
## Description

The current action workflow skips cause requried
actions to be marked as passing until they fail,
not marked as incomplete until either passed,
failed, or skipped.  This change should provide
the latter behavior.


## Type of change

- [x] 🌻 Feature

## Issue(s)

* #744

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-14 22:23:17 +00:00
Keepers
110e874e5b
rename password to passphrase (#796)
General user friendliness renaming.  Will cause a
breaking change in any local environment for
developers and Ark.
2022-09-12 10:09:59 -06:00
Keepers
35c6f2b6b0
undo auto-success workflow skip (#788) 2022-09-07 14:47:30 -06:00
Keepers
1f8c3cdb11
only run actions on /src changes (#750)
## Description

Re-opening the PR with a hack (sourced from github
themselves) to skip the required check if we don't run
the standard required jobs.

## Type of change

- [x] 🌻 Feature

## Issue(s)

#744

## Test Plan

- [x] 💪 Manual
- [ ]  Unit test
- [ ] 💚 E2E
2022-09-02 22:30:18 +00:00
Keepers
9c62134cb2
change command folding in cleanup yml (#546) 2022-08-15 10:36:07 -06:00
Keepers
20aa3d3a28
purge test user folders every half hour (#491) 2022-08-05 18:24:58 +00:00
Keepers
9326690338
autopopulate cli markdown on ci build (#453) 2022-08-04 16:46:14 -06:00
Keepers
dcbc149aa7
Build binary test separately (#399)
* refactor ci caching to ensure go test runs without downloading packages
2022-07-25 09:05:40 -06:00
ashmrtn
395c7c8525
Add spell check linter and run gofmt in golangci-lint (#396)
* Basic golangci-lint config with gofmt and timeout

Remove corresponding sections from other CI config files.

* Enable comment and string spelling linter

* Fix spelling mistakes for linter

Co-authored-by: Danny <danny@alcion.ai>
2022-07-22 18:28:25 +00:00
Keepers
a823231afd
cache go mod in linter action (#367) 2022-07-20 09:20:22 -06:00
Danny
d00332f328
GC async population of m365 objects mail (#332)
-- Changes made to incorporate async population of a DataCollection
2022-07-18 21:01:58 +00:00
Danny
ecdfbbc1a8
CI Cache Implementation (#354)
Caching adding for build and mods. Significantly improves CI `go test -v ./...`.
2022-07-15 16:54:24 -04:00
Danny
76a5d6bba3
Issue #15 adding lint to CI Pipeline. (#325)
Added golangci-lint run check to repo. Currently enables only the defaults.

https://golangci-lint.run/usage/linters
2022-07-13 22:59:58 +00:00
Keepers
2104e66fea
add graph connector to backupOp (#165)
Adds graph connector to the backup operation implementation.
This includes feeding the account credentials into the
backup as well.
2022-06-08 17:32:14 -06:00
Vaibhav Kamra
a91a739d0e
Only run CI once for pull requests (#169)
Currently CI is triggered twice when a PR is updated (the second
run being triggered by the branch push). This removes that
trigger.
2022-06-08 15:45:06 -06:00
Keepers
44392ab8b2
adds env-based integration test controls (#128)
* adds env-based integration test controls

Some integration tests were running locally without being told
to do so via env variables.  This adds some controllers for validating
if and when those tests should run, and for ensuring the env
variables necessary to run the test are available.

* testing helper comments, func name change

* add testing timeOf log, skip gc ci tests

* corrected the test skip in GC_test

* update kopia_test with helpers

* rename /testing files to *_test.go

This will ensure these files do not compile into the release binary.

* Revert "rename /testing files to *_test.go"

This reverts commit 04fd2046cc6d11bf3e3eb98d98a3e84b1bba6366.
2022-06-06 13:17:40 -06:00
Keepers
535cb9e1f5
add common config and encryption passwd (#74) (#99)
* add common config and encryption passwd

Adds a provider-independent configuration handler, and the
the encryption password config property.  The password is used
to encrypt and decrypt the kopia repository properties file.

* fix corso_password in ci.yml

* actually use the corso password in testing

* replace passwd in ci.yml with a secret

* ci.yml secret typo fix
2022-06-01 16:00:41 -06:00
Keepers
596a4cc37b
initial repo init integration test (#90) (#96)
Implements an integration test in the repository that runs an end-
to-end test of an s3 repo initialization through kopia.
2022-05-31 18:32:26 -06:00
Vaibhav Kamra
1e65e2f120
Add go-licenses check to CI (#50)
Runs github.com/google/go-licenses
2022-05-18 14:19:53 -07:00
Vaibhav Kamra
ee37e62927
Add gofmt to ci steps (#16)
This commit has the following changes:

- Rename unitTests.yml to ci.yml
- Add a step to run gofmt
2022-05-09 10:47:43 -07:00