larryh/corso
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rework build.sh script (#1167)

Browse Source 
## Description

While trying to work on https://github.com/alcionai/corso/issues/1166, I ended up reworking parts of the build script ~, but I'm running into some issues with building it on CI. Pushing it here just for reference.~

The new one combines both `build.sh` and `build-container.sh` into a single script where you can specify what to build. Also, inorder setup a proper multi arch, build system locally, we will have to properly setup buildx which is explained in https://docs.docker.com/build/building/multi-platform/ or https://stackoverflow.com/a/70837025/2724649 . I'll add this instructions to docs.

The new build script looks something like this:

```
Usage: build.sh <binary|image> [--platforms ...] [--tag ...]

OPTIONS
 -p|--platforms  Platforms to build for (default: linux/amd64)
                 Specify multiple platforms using ',' (eg: linux/amd64,darwin/arm)
 -t|--tag        Tag for container image (default: alcionai/corso)
```

---
I've made sure the image and binary has the proper architecure and that the amd64 one runs properly in my system. It would be helpful if someone who has access to arm system can validate the arm image. You can use https://github.com/alcionai/corso/pkgs/container/corso/45878348?tag=84fc9d4 image to verify.

```
$ cat check-image.sh
imgid="$(docker create "$1")"
docker cp "$imgid:corso" /tmp/corso

echo Image: "$(docker inspect "$1" | jq '.[0].Architecture')"
echo Binary: "$(file /tmp/corso)"

$ ./check-image.sh ghcr.io/alcionai/corso:84fc9d4@sha256:2278a2b4f108e5dd2ae545f53da1d151b77171f969ffd9718e4bb9886e332ee2
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
Image: "arm64"
Binary: /tmp/corso: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, Go BuildID=7iTBXW0reyfIOS-b-ciS/a5K0Q1IjuA0m9DJxmuNk/Ju1lI6bUZeKn6M_xqon6/KNXwYSnL7e5RVtjAKW9A, not stripped

$ ./check-image.sh ghcr.io/alcionai/corso:84fc9d4@sha256:6320b95470014ca07b9cf1db98b73f5672870c2c53c22c3d13223d88fa621ee0
Image: "amd64"
Binary: /tmp/corso: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, Go BuildID=sBziMHPFI9K-G0et0WjJ/Pe9A2Vy8_xpV3FEDJUMo/p4UeMEzgheASvylZ1N3j/fnwmDeVif4rhneou-S6O, not stripped

$ docker run -it --rm ghcr.io/alcionai/corso:84fc9d4@sha256:6320b95470014ca07b9cf1db98b73f5672870c2c53c22c3d13223d88fa621ee0
[...help message...]
```

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1166

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
This commit is contained in:
Abin Simon 2022-10-18 10:58:44 +05:30 committed by GitHub
parent 4be8e825ad
commit a9acbc28f5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 120 additions and 270 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
.github/workflows/ci.yml vendored
View File

@ -1,6 +1,6 @@
name: Build/Release Corso name: Build/Release Corso
on: on:
workflow_dispatch: # TODO(meain): post-merge: verify manual dispatch workflow_dispatch:
pull_request: pull_request:
branches: [ main ] branches: [ main ]
push: push:
@ -238,7 +238,7 @@ jobs:
run: echo "::set-output name=version::$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)" run: echo "::set-output name=version::$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)"
- name: Run GoReleaser - name: Run GoReleaser
uses: goreleaser/goreleaser-action@v3 # TODO(meain): make sure release builds work uses: goreleaser/goreleaser-action@v3
with: with:
version: latest version: latest
args: release --rm-dist --timeout 500m args: release --rm-dist --timeout 500m
@ -285,7 +285,6 @@ jobs:
npm ci npm ci
CORSO_DOCS_BASEURL="/preview/" npm run build # TODO: update base url once finalized CORSO_DOCS_BASEURL="/preview/" npm run build # TODO: update base url once finalized
# TODO(meain): post-merge: validate push to prod env
- name: Push docs - name: Push docs
run: | run: |
echo "$DOCS_BUCKET" | base64 echo "$DOCS_BUCKET" | base64
@ -309,20 +308,10 @@ jobs:
steps: steps:
- uses: actions/checkout@v3 - uses: actions/checkout@v3
- name: Build Corso Binaries # Setup buildx
run: |
export CORSO_BUILD_LDFLAGS="-X 'github.com/alcionai/corso/src/internal/events.RudderStackWriteKey=${{ secrets.RUDDERSTACK_CORSO_WRITE_KEY }}' \
-X 'github.com/alcionai/corso/src/internal/events.RudderStackDataPlaneURL=${{ secrets.RUDDERSTACK_CORSO_DATA_PLANE_URL }}' \
-X 'github.com/alcionai/corso/src/cli.version=$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)'"
./build.sh --platforms ${{ env.PLATFORMS }}
# apparently everyone uses this step
- name: Set up QEMU - name: Set up QEMU
uses: docker/setup-qemu-action@v2 uses: docker/setup-qemu-action@v2
# setup Docker build action
- name: Set up Docker Buildx - name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2 uses: docker/setup-buildx-action@v2
# retrieve credentials for ghcr.io # retrieve credentials for ghcr.io
@ -337,7 +326,7 @@ jobs:
id: meta id: meta
uses: docker/metadata-action@v4 uses: docker/metadata-action@v4
with: with:
images: ${{ env.imageName }} # TODO(meain): post-merge: validate push with tag images: ${{ env.imageName }}
tags: | tags: |
type=ref,event=tag type=ref,event=tag
type=sha,format=short,prefix= type=sha,format=short,prefix=
@ -352,6 +341,8 @@ jobs:
push: true push: true
tags: ${{ steps.meta.outputs.tags }} tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }} labels: ${{ steps.meta.outputs.labels }}
build-args: |
CORSO_BUILD_LDFLAGS=-X 'github.com/alcionai/corso/src/internal/events.RudderStackWriteKey=${{ secrets.RUDDERSTACK_CORSO_WRITE_KEY }}' -X 'github.com/alcionai/corso/src/internal/events.RudderStackDataPlaneURL=${{ secrets.RUDDERSTACK_CORSO_DATA_PLANE_URL }}' -X 'github.com/alcionai/corso/src/cli.version=$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)'
# use the github cache # use the github cache
cache-from: type=gha cache-from: type=gha
cache-to: type=gha,mode=max cache-to: type=gha,mode=max

37
build/Dockerfile
View File

@ -1,34 +1,31 @@
# syntax=docker/dockerfile:1 FROM golang:1.18-alpine as builder
# This dockerfile is configured to be run by the /corso/build/build-container.sh WORKDIR /go/src/app
# script. Using docker to build this file directly will fail. COPY src .
ARG CORSO_BUILD_LDFLAGS="" # ldflags
RUN go build -o corso -ldflags "$CORSO_BUILD_LDFLAGS"
FROM alpine:3.16 FROM alpine:3.16
ENV CORSO_HOME /app/corso LABEL org.opencontainers.image.title="Corso"
LABEL org.opencontainers.image.description="Free, Secure, and Open-Source Backup for Microsoft 365"
LABEL org.opencontainers.image.url="https://github.com/alcionai/corso"
LABEL org.opencontainers.image.source="https://github.com/alcionai/corso"
LABEL org.opencontainers.image.vendor="Alcion, Inc."
# Add a well-know corso user to use by default COPY --from=builder /go/src/app/corso /corso
RUN addgroup -g 1001 corso && \
adduser --shell /sbin/nologin --disabled-password \
--home $CORSO_HOME --uid 1001 --ingroup corso corso
# Update to latest cert bundle in case there are changes # Pull tls certs directly from latest upstream image
RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/* COPY --from=alpine:latest /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
# Default locations for key Kopia config as set in https://github.com/kopia/kopia/blob/master/tools/docker/Dockerfile ENV CORSO_HOME=/app/corso
ENV KOPIA_CONFIG_PATH=$CORSO_HOME/kopia/config/repository.config \ ENV CORSO_CONFIG_DIR=$CORSO_HOME \
KOPIA_CONFIG_PATH=$CORSO_HOME/kopia/config/repository.config \
KOPIA_LOG_DIR=$CORSO_HOME/kopia/logs \ KOPIA_LOG_DIR=$CORSO_HOME/kopia/logs \
KOPIA_CACHE_DIRECTORY=$CORSO_HOME/kopia/cache \ KOPIA_CACHE_DIRECTORY=$CORSO_HOME/kopia/cache \
RCLONE_CONFIG=$CORSO_HOME/kopia/rclone/rclone.conf \ RCLONE_CONFIG=$CORSO_HOME/kopia/rclone/rclone.conf \
KOPIA_PERSIST_CREDENTIALS_ON_CONNECT=false \ KOPIA_PERSIST_CREDENTIALS_ON_CONNECT=false \
KOPIA_CHECK_FOR_UPDATES=false KOPIA_CHECK_FOR_UPDATES=false
WORKDIR /
ARG TARGETOS
ARG TARGETARCH
COPY ./bin/${TARGETOS}-${TARGETARCH}/corso ./
USER corso
ENTRYPOINT ["/corso"] ENTRYPOINT ["/corso"]

100
build/build-container.sh
View File

@ -1,100 +0,0 @@
#!/bin/sh
set -e
usage() {
echo " "
echo "-----"
echo "Builds a Corso docker container image."
echo " "
echo "-----"
echo "Flags"
echo " -h|--help Help"
echo " -a|--arch Set the architecture to the specified value (default: amd64)"
echo " -l|--local Build the corso binary on your local system, rather than a go image"
echo " -p|--prefix Prefixes the image name."
echo " -s|--suffix Suffixes the version."
echo " "
echo "-----"
echo "Example Usage:"
echo " ./build/build-container.sh"
echo " ./build/build-container.sh --arch arm64"
echo " ./build/build-container.sh --arch arm64 --prefix ghcr.io --suffix nightly"
echo " "
exit 0
}
SCRIPT_ROOT=$(dirname $(readlink -f $0))
PROJECT_ROOT=$(dirname ${SCRIPT_ROOT})
OS=linux
ARCH=amd64
IMAGE_NAME_PREFIX=
IMAGE_TAG_SUFFIX=
LOCAL=
while [ "$#" -gt 0 ]
do
case "$1" in
-h|--help)
usage
exit 0
;;
-a|--arch)
ARCH=$2
shift
;;
-l|--local)
LOCAL=1
;;
-p|--prefix)
IMAGE_NAME_PREFIX=$2
shift
;;
-s|--suffix)
IMAGE_TAG_SUFFIX=$2
shift
;;
-*)
echo "Invalid flag '$1'. Use -h|--help to see the valid options" >&2
return 1
;;
*)
echo "Invalid arg '$1'. Use -h|--help to see the valid options" >&2
return 1
;;
esac
shift
done
TARGETPLATFORM=${OS}/${ARCH}
IMAGE_TAG=${OS}-${ARCH}
if [ ! -z "${IMAGE_TAG_SUFFIX}" ]; then
IMAGE_TAG=${IMAGE_TAG}-${IMAGE_TAG_SUFFIX}
fi
IMAGE_NAME=alcionai/corso:${IMAGE_TAG}
if [ ! -z "${IMAGE_NAME_PREFIX}" ]; then
IMAGE_NAME=${IMAGE_NAME_PREFIX}/${IMAGE_NAME}
fi
if [ -z "$LOCAL" ]; then
${SCRIPT_ROOT}/build.sh --platforms "${TARGETPLATFORM}"
else
${SCRIPT_ROOT}/multiplatform-binary.sh --platforms "${TARGETPLATFORM}"
fi
echo "-----"
echo "building corso container ${IMAGE_NAME}"
echo "-----"
set -x
docker buildx build --tag ${IMAGE_NAME} \
--platform ${TARGETPLATFORM} \
--file ${PROJECT_ROOT}/build/Dockerfile \
${PROJECT_ROOT}
set +x
echo "-----"
echo "container built successfully"

118
build/build.sh
View File

@ -2,74 +2,80 @@
set -e set -e
SCRIPT_ROOT=$(dirname $(readlink -f $0)) ROOT=$(dirname $(dirname $(readlink -f $0)))
PROJECT_ROOT=$(dirname ${SCRIPT_ROOT}) GOVER=1.18 # go version
CORSO_BUILD_CACHE="/tmp/.corsobuild" # shared persistent cache
CORSO_BUILD_CONTAINER=/go/src/github.com/alcionai/corso # Figure out os and architecture
CORSO_BUILD_CONTAINER_SRC=${CORSO_BUILD_CONTAINER}/src case "$(uname -m)" in
CORSO_BUILD_PKG_MOD=/go/pkg/mod x86_64) GOARCH="amd64" ;;
CORSO_BUILD_TMP=/tmp/.corsobuild aarch64) GOARCH="arm64" ;;
CORSO_BUILD_TMP_CACHE=${CORSO_BUILD_TMP}/cache arm) GOARCH="arm" ;;
CORSO_BUILD_TMP_MOD=${CORSO_BUILD_TMP}/mod i386) GOARCH="386" ;;
CORSO_CACHE=${CORSO_BUILD_TMP_CACHE} *) echo "Unknown architecture" && exit 0 ;;
CORSO_MOD_CACHE=${CORSO_BUILD_PKG_MOD}/cache esac
case "$(uname)" in
Linux) GOOS="linux" ;;
Darwin) GOOS="darwin" ;; # TODO: verify this
*) echo "Unknown OS" && exit 0 ;;
esac
CORSO_BUILD_ARGS='' PLATFORMS="$GOOS/$GOARCH" # default platform
TAG="alcionai/corso" # default image tag
platforms= usage() {
GOVER=1.18 echo "Usage: $(basename $0) <binary|image> [--platforms ...] [--tag ...]"
GOOS=linux echo ""
GOARCH=amd64 echo "OPTIONS"
echo " -p|--platforms Platforms to build for (default: $PLATFORMS)"
echo " Specify multiple platforms using ',' (eg: linux/amd64,darwin/arm)"
echo " -t|--tag Tag for container image (default: $TAG)"
}
while [ "$#" -gt 0 ] MODE="binary"
do case "$1" in
binary) MODE="binary" && shift ;;
image) MODE="image" && shift ;;
-h | --help) usage && exit 0 ;;
*) usage && exit 1 ;;
esac
while [ "$#" -gt 0 ]; do
case "$1" in case "$1" in
--platforms) -p | --platforms) PLATFORMS="$2" && shift ;;
platforms=$2 -t | --tag) TAG="$2" && shift ;;
shift *) echo "Invalid argument $1" && usage && exit 1 ;;
;;
esac esac
shift shift
done done
# temporary directory for caching go build if [ "$MODE" == "binary" ]; then
mkdir -p ${CORSO_BUILD_TMP_CACHE} mkdir -p ${CORSO_BUILD_CACHE} # prep env
# temporary directory for caching go modules (needed for fast cross-platform build) for platform in ${PLATFORMS/,/ }; do
mkdir -p ${CORSO_BUILD_TMP_MOD} IFS='/' read -r -a platform_split <<<"$platform"
if [ -z "$platforms" ]; then
platforms="${GOOS}/${GOARCH}"
fi
for platform in ${platforms/,/ }
do
IFS='/' read -r -a platform_split <<< "${platform}"
GOOS=${platform_split[0]} GOOS=${platform_split[0]}
GOARCH=${platform_split[1]} GOARCH=${platform_split[1]}
echo "-----" printf "Building for %s...\r" "$platform"
echo "building corso binary for ${GOOS}/${GOARCH}"
echo "-----"
set -x
docker run --rm \ docker run --rm \
--mount type=bind,src=${PROJECT_ROOT},dst=${CORSO_BUILD_CONTAINER} \ --mount type=bind,src=${ROOT},dst="/app" \
--mount type=bind,src=${CORSO_BUILD_TMP_CACHE},dst=${CORSO_BUILD_TMP_CACHE} \ --mount type=bind,src=${CORSO_BUILD_CACHE},dst=${CORSO_BUILD_CACHE} \
--mount type=bind,src=${CORSO_BUILD_TMP_MOD},dst=${CORSO_BUILD_PKG_MOD} \ --env GOMODCACHE=${CORSO_BUILD_CACHE}/mod --env GOCACHE=${CORSO_BUILD_CACHE}/cache \
--workdir ${CORSO_BUILD_CONTAINER_SRC} \ --env GOOS=${GOOS} --env GOARCH=${GOARCH} \
--env GOMODCACHE=${CORSO_MOD_CACHE} \ --workdir "/app/src" \
--env GOCACHE=${CORSO_CACHE} \
--env GOOS=${GOOS} \
--env GOARCH=${GOARCH} \
--entrypoint /usr/local/go/bin/go \
golang:${GOVER} \ golang:${GOVER} \
build -o corso ${CORSO_BUILD_ARGS} -ldflags "${CORSO_BUILD_LDFLAGS}" go build -o corso -ldflags "${CORSO_BUILD_LDFLAGS}"
set +x
mkdir -p ${PROJECT_ROOT}/bin/${GOOS}-${GOARCH} mkdir -p ${ROOT}/bin/${GOOS}-${GOARCH}
mv ${PROJECT_ROOT}/src/corso ${PROJECT_ROOT}/bin/${GOOS}-${GOARCH}/corso mv ${ROOT}/src/corso ${ROOT}/bin/${GOOS}-${GOARCH}/corso
echo Corso $platform binary available in ${ROOT}/bin/${GOOS}-${GOARCH}/corso
echo "-----" done
echo "created binary image in ${PROJECT_ROOT}/bin/${GOOS}-${GOARCH}/corso" else
echo "-----" echo Building "$TAG" image for "$PLATFORMS"
done docker buildx build --tag ${TAG} \
--platform ${PLATFORMS} \
--file ${ROOT}/build/Dockerfile \
--build-arg CORSO_BUILD_LDFLAGS="$CORSO_BUILD_LDFLAGS" \
--load ${ROOT}
echo Built container image "$TAG"
fi

56
build/multiplatform-binary.sh
View File

@ -1,56 +0,0 @@
#!/bin/bash
set -e
SCRIPT_ROOT=$(dirname $(readlink -f $0))
PROJECT_ROOT=$(dirname ${SCRIPT_ROOT})
platforms=
GOVER=1.18
GOOS=linux
GOARCH=amd64
while [ "$#" -gt 0 ]
do
case "$1" in
--platforms)
platforms=$2
shift
;;
esac
shift
done
CORSO_BUILD_ARGS="$@"
if [ -z "$platforms" ]; then
platforms="${GOOS}/${GOARCH}"
fi
for platform in ${platforms/,/ }
do
IFS='/' read -r -a platform_split <<< "${platform}"
GOOS=${platform_split[0]}
GOARCH=${platform_split[1]}
echo "-----"
echo "building corso binary for ${GOOS}/${GOARCH}"
echo "-----"
OS_ARCH_DIR=${PROJECT_ROOT}/bin/${GOOS}-${GOARCH}
set -x
mkdir -p ${OS_ARCH_DIR}
cd ${PROJECT_ROOT}/src; \
GOOS=${GOOS} \
GOARCH=${GOARCH} \
go build -o ${OS_ARCH_DIR}/corso $CORSO_BUILD_ARGS
set +x
echo "-----"
echo "created binary ${PROJECT_ROOT}/bin/${GOOS}-${GOARCH}/corso"
echo "-----"
done

42
docs/docs/developers/build.md
View File

@ -13,7 +13,7 @@ go build -o corso
``` ```
:::info :::info
If you dong have Go available, you can find installation instructions [here](https://go.dev/doc/install) If you don't have Go available, you can find installation instructions [here](https://go.dev/doc/install)
::: :::
This will generate a binary named `corso` in the directory where you run the build. This will generate a binary named `corso` in the directory where you run the build.
@ -26,16 +26,17 @@ For convenience, the Corso build tooling is containerized. To take advantage, yo
To build Corso via docker, use the following command from the root of your repo: To build Corso via docker, use the following command from the root of your repo:
```bash ```bash
./build/build.sh ./build/build.sh binary
``` ```
You can pass in all the architectures/platforms you would like to By default, we will build for your current platform. You can pass in
build it for using the `--platforms` flag as a comma separated all the architectures/platforms you would like to build it for using
list. For example, if you would like to build `amd64` and `arm64` the `--platforms` flag as a comma separated list. For example, if you
versions for Linux, you can run the following command: would like to build `amd64` and `arm64` versions for Linux, you can
run the following command:
```bash ```bash
./build/build.sh --platforms linux/amd64,linux/arm64 ./build/build.sh binary --platforms linux/amd64,linux/arm64
``` ```
Once built, the resulting binaries will be available in `<repo root>/bin` for all the different platforms you specified. Once built, the resulting binaries will be available in `<repo root>/bin` for all the different platforms you specified.
@ -46,18 +47,29 @@ If you prefer to build Corso as a container image, use the following command ins
```bash ```bash
# Use --help to see all available options # Use --help to see all available options
./build/build-container.sh ./build/build.sh image
``` ```
Below are the main customization flags that you can set when building a container image: :::note
`Dockerfile` used to build the image is available at [`build/Dockerfile`](https://github.com/alcionai/corso/blob/main/build/Dockerfile)
:::
- `-a|--arch`: Set the architecture to the specified value. (default: amd64) Similar to binaries, we build your a container image for your current
- `-l|--local`: Build the corso binary on your local system, rather than a go image. platform by default, but you can change it by explicitly passing in
- `-p|--prefix`: Prefix for the image name. the platforms that you would like to build for.
- `-s|--suffix`: Suffix for the version. In addition, you can optionally pass the tag that you would like to
apply for the image using `--tag` option.
For example, you can use the following command to create a `arm64` image with prefix of `ghcr.io` and the tag as `nightly`. For example, you can use the following command to create a `arm64`
image with the tag `gcr.io/alcionai/corso:latest`, you can run:
```bash ```bash
./build/build-container.sh --arch arm64 --prefix ghcr.io --suffix nightly ./build/build.sh image --platforms linux/arm64 --tag gcr.io/alcionai/corso:latest
``` ```
:::info
If you run into any issues with building cross platform images, make
sure to follow the instructions on [Docker
docs](https://docs.docker.com/build/building/multi-platform/) to setup
the build environment for Multi-platform images.
:::