larryh/corso
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rework build.sh script (#1167)

Browse Source 
## Description

While trying to work on https://github.com/alcionai/corso/issues/1166, I ended up reworking parts of the build script ~, but I'm running into some issues with building it on CI. Pushing it here just for reference.~

The new one combines both `build.sh` and `build-container.sh` into a single script where you can specify what to build. Also, inorder setup a proper multi arch, build system locally, we will have to properly setup buildx which is explained in https://docs.docker.com/build/building/multi-platform/ or https://stackoverflow.com/a/70837025/2724649 . I'll add this instructions to docs.

The new build script looks something like this:

```
Usage: build.sh <binary|image> [--platforms ...] [--tag ...]

OPTIONS
 -p|--platforms  Platforms to build for (default: linux/amd64)
                 Specify multiple platforms using ',' (eg: linux/amd64,darwin/arm)
 -t|--tag        Tag for container image (default: alcionai/corso)
```

---
I've made sure the image and binary has the proper architecure and that the amd64 one runs properly in my system. It would be helpful if someone who has access to arm system can validate the arm image. You can use https://github.com/alcionai/corso/pkgs/container/corso/45878348?tag=84fc9d4 image to verify.

```
$ cat check-image.sh
imgid="$(docker create "$1")"
docker cp "$imgid:corso" /tmp/corso

echo Image: "$(docker inspect "$1" | jq '.[0].Architecture')"
echo Binary: "$(file /tmp/corso)"

$ ./check-image.sh ghcr.io/alcionai/corso:84fc9d4@sha256:2278a2b4f108e5dd2ae545f53da1d151b77171f969ffd9718e4bb9886e332ee2
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
Image: "arm64"
Binary: /tmp/corso: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, Go BuildID=7iTBXW0reyfIOS-b-ciS/a5K0Q1IjuA0m9DJxmuNk/Ju1lI6bUZeKn6M_xqon6/KNXwYSnL7e5RVtjAKW9A, not stripped

$ ./check-image.sh ghcr.io/alcionai/corso:84fc9d4@sha256:6320b95470014ca07b9cf1db98b73f5672870c2c53c22c3d13223d88fa621ee0
Image: "amd64"
Binary: /tmp/corso: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, Go BuildID=sBziMHPFI9K-G0et0WjJ/Pe9A2Vy8_xpV3FEDJUMo/p4UeMEzgheASvylZ1N3j/fnwmDeVif4rhneou-S6O, not stripped

$ docker run -it --rm ghcr.io/alcionai/corso:84fc9d4@sha256:6320b95470014ca07b9cf1db98b73f5672870c2c53c22c3d13223d88fa621ee0
[...help message...]
```

## Type of change

<!--- Please check the type of change your PR introduces: --->
- [ ] 🌻 Feature
- [ ] 🐛 Bugfix
- [ ] 🗺️ Documentation
- [ ] 🤖 Test
- [x] 💻 CI/Deployment
- [ ] 🐹 Trivial/Minor

## Issue(s)

<!-- Can reference multiple issues. Use one of the following "magic words" - "closes, fixes" to auto-close the Github issue. -->
* https://github.com/alcionai/corso/issues/1166

## Test Plan

<!-- How will this be tested prior to merging.-->
- [x] 💪 Manual
- [ ] ⚡ Unit test
- [ ] 💚 E2E
This commit is contained in:
Abin Simon 2022-10-18 10:58:44 +05:30 committed by GitHub
parent 4be8e825ad
commit a9acbc28f5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 120 additions and 270 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
.github/workflows/ci.yml vendored
View File

@ -1,6 +1,6 @@
name: Build/Release Corso
on:
workflow_dispatch: # TODO(meain): post-merge: verify manual dispatch
workflow_dispatch:
pull_request:
branches: [ main ]
push:
@ -238,7 +238,7 @@ jobs:
run: echo "::set-output name=version::$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)"
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@v3 # TODO(meain): make sure release builds work
uses: goreleaser/goreleaser-action@v3
with:
version: latest
args: release --rm-dist --timeout 500m
@ -285,7 +285,6 @@ jobs:
npm ci
CORSO_DOCS_BASEURL="/preview/" npm run build # TODO: update base url once finalized
# TODO(meain): post-merge: validate push to prod env
- name: Push docs
run: |
echo "$DOCS_BUCKET" | base64
@ -309,20 +308,10 @@ jobs:
steps:
- uses: actions/checkout@v3
- name: Build Corso Binaries
run: |
export CORSO_BUILD_LDFLAGS="-X 'github.com/alcionai/corso/src/internal/events.RudderStackWriteKey=${{ secrets.RUDDERSTACK_CORSO_WRITE_KEY }}' \
-X 'github.com/alcionai/corso/src/internal/events.RudderStackDataPlaneURL=${{ secrets.RUDDERSTACK_CORSO_DATA_PLANE_URL }}' \
-X 'github.com/alcionai/corso/src/cli.version=$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)'"
./build.sh --platforms ${{ env.PLATFORMS }}
# apparently everyone uses this step
# Setup buildx
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
# setup Docker build action
- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v2
# retrieve credentials for ghcr.io
@ -337,7 +326,7 @@ jobs:
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.imageName }} # TODO(meain): post-merge: validate push with tag
images: ${{ env.imageName }}
tags: |
type=ref,event=tag
type=sha,format=short,prefix=
@ -352,6 +341,8 @@ jobs:
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
CORSO_BUILD_LDFLAGS=-X 'github.com/alcionai/corso/src/internal/events.RudderStackWriteKey=${{ secrets.RUDDERSTACK_CORSO_WRITE_KEY }}' -X 'github.com/alcionai/corso/src/internal/events.RudderStackDataPlaneURL=${{ secrets.RUDDERSTACK_CORSO_DATA_PLANE_URL }}' -X 'github.com/alcionai/corso/src/cli.version=$(git describe --exact-match --tags $(git rev-parse HEAD) 2>/dev/null || echo unreleased)-$(git rev-parse --short HEAD)'
# use the github cache
cache-from: type=gha
cache-to: type=gha,mode=max

37
build/Dockerfile
View File

@ -1,34 +1,31 @@
# syntax=docker/dockerfile:1
FROM golang:1.18-alpine as builder
# This dockerfile is configured to be run by the /corso/build/build-container.sh
# script. Using docker to build this file directly will fail.
WORKDIR /go/src/app
COPY src .
ARG CORSO_BUILD_LDFLAGS="" # ldflags
RUN go build -o corso -ldflags "$CORSO_BUILD_LDFLAGS"
FROM alpine:3.16
ENV CORSO_HOME /app/corso
LABEL org.opencontainers.image.title="Corso"
LABEL org.opencontainers.image.description="Free, Secure, and Open-Source Backup for Microsoft 365"
LABEL org.opencontainers.image.url="https://github.com/alcionai/corso"
LABEL org.opencontainers.image.source="https://github.com/alcionai/corso"
LABEL org.opencontainers.image.vendor="Alcion, Inc."
# Add a well-know corso user to use by default
RUN addgroup -g 1001 corso && \
adduser --shell /sbin/nologin --disabled-password \
--home $CORSO_HOME --uid 1001 --ingroup corso corso
COPY --from=builder /go/src/app/corso /corso
# Update to latest cert bundle in case there are changes
RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/*
# Pull tls certs directly from latest upstream image
COPY --from=alpine:latest /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
# Default locations for key Kopia config as set in https://github.com/kopia/kopia/blob/master/tools/docker/Dockerfile
ENV KOPIA_CONFIG_PATH=$CORSO_HOME/kopia/config/repository.config \
ENV CORSO_HOME=/app/corso
ENV CORSO_CONFIG_DIR=$CORSO_HOME \
KOPIA_CONFIG_PATH=$CORSO_HOME/kopia/config/repository.config \
KOPIA_LOG_DIR=$CORSO_HOME/kopia/logs \
KOPIA_CACHE_DIRECTORY=$CORSO_HOME/kopia/cache \
RCLONE_CONFIG=$CORSO_HOME/kopia/rclone/rclone.conf \
KOPIA_PERSIST_CREDENTIALS_ON_CONNECT=false \
KOPIA_CHECK_FOR_UPDATES=false
WORKDIR /
ARG TARGETOS
ARG TARGETARCH
COPY ./bin/${TARGETOS}-${TARGETARCH}/corso ./
USER corso
ENTRYPOINT ["/corso"]

100
build/build-container.sh
View File

@ -1,100 +0,0 @@
#!/bin/sh
set -e
usage() {
echo " "
echo "-----"
echo "Builds a Corso docker container image."
echo " "
echo "-----"
echo "Flags"
echo " -h|--help Help"
echo " -a|--arch Set the architecture to the specified value (default: amd64)"
echo " -l|--local Build the corso binary on your local system, rather than a go image"
echo " -p|--prefix Prefixes the image name."
echo " -s|--suffix Suffixes the version."
echo " "
echo "-----"
echo "Example Usage:"
echo " ./build/build-container.sh"
echo " ./build/build-container.sh --arch arm64"
echo " ./build/build-container.sh --arch arm64 --prefix ghcr.io --suffix nightly"
echo " "
exit 0
}
SCRIPT_ROOT=$(dirname $(readlink -f $0))
PROJECT_ROOT=$(dirname ${SCRIPT_ROOT})
OS=linux
ARCH=amd64
IMAGE_NAME_PREFIX=
IMAGE_TAG_SUFFIX=
LOCAL=
while [ "$#" -gt 0 ]
do
case "$1" in
-h|--help)
usage
exit 0
;;
-a|--arch)
ARCH=$2
shift
;;
-l|--local)
LOCAL=1
;;
-p|--prefix)
IMAGE_NAME_PREFIX=$2
shift
;;
-s|--suffix)
IMAGE_TAG_SUFFIX=$2
shift
;;
-*)
echo "Invalid flag '$1'. Use -h|--help to see the valid options" >&2
return 1
;;
*)
echo "Invalid arg '$1'. Use -h|--help to see the valid options" >&2
return 1
;;
esac
shift
done
TARGETPLATFORM=${OS}/${ARCH}
IMAGE_TAG=${OS}-${ARCH}
if [ ! -z "${IMAGE_TAG_SUFFIX}" ]; then
IMAGE_TAG=${IMAGE_TAG}-${IMAGE_TAG_SUFFIX}
fi
IMAGE_NAME=alcionai/corso:${IMAGE_TAG}
if [ ! -z "${IMAGE_NAME_PREFIX}" ]; then
IMAGE_NAME=${IMAGE_NAME_PREFIX}/${IMAGE_NAME}
fi
if [ -z "$LOCAL" ]; then
${SCRIPT_ROOT}/build.sh --platforms "${TARGETPLATFORM}"
else
${SCRIPT_ROOT}/multiplatform-binary.sh --platforms "${TARGETPLATFORM}"
fi
echo "-----"
echo "building corso container ${IMAGE_NAME}"
echo "-----"
set -x
docker buildx build --tag ${IMAGE_NAME} \
--platform ${TARGETPLATFORM} \
--file ${PROJECT_ROOT}/build/Dockerfile \
${PROJECT_ROOT}
set +x
echo "-----"
echo "container built successfully"

132
build/build.sh
View File

@ -2,74 +2,80 @@
set -e
SCRIPT_ROOT=$(dirname $(readlink -f $0))
PROJECT_ROOT=$(dirname ${SCRIPT_ROOT})
ROOT=$(dirname $(dirname $(readlink -f $0)))
GOVER=1.18 # go version
CORSO_BUILD_CACHE="/tmp/.corsobuild" # shared persistent cache
CORSO_BUILD_CONTAINER=/go/src/github.com/alcionai/corso
CORSO_BUILD_CONTAINER_SRC=${CORSO_BUILD_CONTAINER}/src
CORSO_BUILD_PKG_MOD=/go/pkg/mod
CORSO_BUILD_TMP=/tmp/.corsobuild
CORSO_BUILD_TMP_CACHE=${CORSO_BUILD_TMP}/cache
CORSO_BUILD_TMP_MOD=${CORSO_BUILD_TMP}/mod
CORSO_CACHE=${CORSO_BUILD_TMP_CACHE}
CORSO_MOD_CACHE=${CORSO_BUILD_PKG_MOD}/cache
# Figure out os and architecture
case "$(uname -m)" in
x86_64) GOARCH="amd64" ;;
aarch64) GOARCH="arm64" ;;
arm) GOARCH="arm" ;;
i386) GOARCH="386" ;;
*) echo "Unknown architecture" && exit 0 ;;
esac
case "$(uname)" in
Linux) GOOS="linux" ;;
Darwin) GOOS="darwin" ;; # TODO: verify this
*) echo "Unknown OS" && exit 0 ;;
esac
CORSO_BUILD_ARGS=''
PLATFORMS="$GOOS/$GOARCH" # default platform
TAG="alcionai/corso" # default image tag
platforms=
GOVER=1.18
GOOS=linux
GOARCH=amd64
usage() {
echo "Usage: $(basename $0) <binary|image> [--platforms ...] [--tag ...]"
echo ""
echo "OPTIONS"
echo " -p|--platforms Platforms to build for (default: $PLATFORMS)"
echo " Specify multiple platforms using ',' (eg: linux/amd64,darwin/arm)"
echo " -t|--tag Tag for container image (default: $TAG)"
}
while [ "$#" -gt 0 ]
do
case "$1" in
--platforms)
platforms=$2
shift
;;
esac
shift
MODE="binary"
case "$1" in
binary) MODE="binary" && shift ;;
image) MODE="image" && shift ;;
-h | --help) usage && exit 0 ;;
*) usage && exit 1 ;;
esac
while [ "$#" -gt 0 ]; do
case "$1" in
-p | --platforms) PLATFORMS="$2" && shift ;;
-t | --tag) TAG="$2" && shift ;;
*) echo "Invalid argument $1" && usage && exit 1 ;;
esac
shift
done
# temporary directory for caching go build
mkdir -p ${CORSO_BUILD_TMP_CACHE}
# temporary directory for caching go modules (needed for fast cross-platform build)
mkdir -p ${CORSO_BUILD_TMP_MOD}
if [ "$MODE" == "binary" ]; then
mkdir -p ${CORSO_BUILD_CACHE} # prep env
for platform in ${PLATFORMS/,/ }; do
IFS='/' read -r -a platform_split <<<"$platform"
GOOS=${platform_split[0]}
GOARCH=${platform_split[1]}
if [ -z "$platforms" ]; then
platforms="${GOOS}/${GOARCH}"
printf "Building for %s...\r" "$platform"
docker run --rm \
--mount type=bind,src=${ROOT},dst="/app" \
--mount type=bind,src=${CORSO_BUILD_CACHE},dst=${CORSO_BUILD_CACHE} \
--env GOMODCACHE=${CORSO_BUILD_CACHE}/mod --env GOCACHE=${CORSO_BUILD_CACHE}/cache \
--env GOOS=${GOOS} --env GOARCH=${GOARCH} \
--workdir "/app/src" \
golang:${GOVER} \
go build -o corso -ldflags "${CORSO_BUILD_LDFLAGS}"
mkdir -p ${ROOT}/bin/${GOOS}-${GOARCH}
mv ${ROOT}/src/corso ${ROOT}/bin/${GOOS}-${GOARCH}/corso
echo Corso $platform binary available in ${ROOT}/bin/${GOOS}-${GOARCH}/corso
done
else
echo Building "$TAG" image for "$PLATFORMS"
docker buildx build --tag ${TAG} \
--platform ${PLATFORMS} \
--file ${ROOT}/build/Dockerfile \
--build-arg CORSO_BUILD_LDFLAGS="$CORSO_BUILD_LDFLAGS" \
--load ${ROOT}
echo Built container image "$TAG"
fi
for platform in ${platforms/,/ }
do
IFS='/' read -r -a platform_split <<< "${platform}"
GOOS=${platform_split[0]}
GOARCH=${platform_split[1]}
echo "-----"
echo "building corso binary for ${GOOS}/${GOARCH}"
echo "-----"
set -x
docker run --rm \
--mount type=bind,src=${PROJECT_ROOT},dst=${CORSO_BUILD_CONTAINER} \
--mount type=bind,src=${CORSO_BUILD_TMP_CACHE},dst=${CORSO_BUILD_TMP_CACHE} \
--mount type=bind,src=${CORSO_BUILD_TMP_MOD},dst=${CORSO_BUILD_PKG_MOD} \
--workdir ${CORSO_BUILD_CONTAINER_SRC} \
--env GOMODCACHE=${CORSO_MOD_CACHE} \
--env GOCACHE=${CORSO_CACHE} \
--env GOOS=${GOOS} \
--env GOARCH=${GOARCH} \
--entrypoint /usr/local/go/bin/go \
golang:${GOVER} \
build -o corso ${CORSO_BUILD_ARGS} -ldflags "${CORSO_BUILD_LDFLAGS}"
set +x
mkdir -p ${PROJECT_ROOT}/bin/${GOOS}-${GOARCH}
mv ${PROJECT_ROOT}/src/corso ${PROJECT_ROOT}/bin/${GOOS}-${GOARCH}/corso
echo "-----"
echo "created binary image in ${PROJECT_ROOT}/bin/${GOOS}-${GOARCH}/corso"
echo "-----"
done

56
build/multiplatform-binary.sh
View File

@ -1,56 +0,0 @@
#!/bin/bash
set -e
SCRIPT_ROOT=$(dirname $(readlink -f $0))
PROJECT_ROOT=$(dirname ${SCRIPT_ROOT})
platforms=
GOVER=1.18
GOOS=linux
GOARCH=amd64
while [ "$#" -gt 0 ]
do
case "$1" in
--platforms)
platforms=$2
shift
;;
esac
shift
done
CORSO_BUILD_ARGS="$@"
if [ -z "$platforms" ]; then
platforms="${GOOS}/${GOARCH}"
fi
for platform in ${platforms/,/ }
do
IFS='/' read -r -a platform_split <<< "${platform}"
GOOS=${platform_split[0]}
GOARCH=${platform_split[1]}
echo "-----"
echo "building corso binary for ${GOOS}/${GOARCH}"
echo "-----"
OS_ARCH_DIR=${PROJECT_ROOT}/bin/${GOOS}-${GOARCH}
set -x
mkdir -p ${OS_ARCH_DIR}
cd ${PROJECT_ROOT}/src; \
GOOS=${GOOS} \
GOARCH=${GOARCH} \
go build -o ${OS_ARCH_DIR}/corso $CORSO_BUILD_ARGS
set +x
echo "-----"
echo "created binary ${PROJECT_ROOT}/bin/${GOOS}-${GOARCH}/corso"
echo "-----"
done

42
docs/docs/developers/build.md
View File

@ -13,7 +13,7 @@ go build -o corso
```
:::info
If you dong have Go available, you can find installation instructions [here](https://go.dev/doc/install)
If you don't have Go available, you can find installation instructions [here](https://go.dev/doc/install)
:::
This will generate a binary named `corso` in the directory where you run the build.
@ -26,16 +26,17 @@ For convenience, the Corso build tooling is containerized. To take advantage, yo
To build Corso via docker, use the following command from the root of your repo:
```bash
./build/build.sh
./build/build.sh binary
```
You can pass in all the architectures/platforms you would like to
build it for using the `--platforms` flag as a comma separated
list. For example, if you would like to build `amd64` and `arm64`
versions for Linux, you can run the following command:
By default, we will build for your current platform. You can pass in
all the architectures/platforms you would like to build it for using
the `--platforms` flag as a comma separated list. For example, if you
would like to build `amd64` and `arm64` versions for Linux, you can
run the following command:
```bash
./build/build.sh --platforms linux/amd64,linux/arm64
./build/build.sh binary --platforms linux/amd64,linux/arm64
```
Once built, the resulting binaries will be available in `<repo root>/bin` for all the different platforms you specified.
@ -46,18 +47,29 @@ If you prefer to build Corso as a container image, use the following command ins
```bash
# Use --help to see all available options
./build/build-container.sh
./build/build.sh image
```
Below are the main customization flags that you can set when building a container image:
:::note
`Dockerfile` used to build the image is available at [`build/Dockerfile`](https://github.com/alcionai/corso/blob/main/build/Dockerfile)
:::
- `-a|--arch`: Set the architecture to the specified value. (default: amd64)
- `-l|--local`: Build the corso binary on your local system, rather than a go image.
- `-p|--prefix`: Prefix for the image name.
- `-s|--suffix`: Suffix for the version.
Similar to binaries, we build your a container image for your current
platform by default, but you can change it by explicitly passing in
the platforms that you would like to build for.
In addition, you can optionally pass the tag that you would like to
apply for the image using `--tag` option.
For example, you can use the following command to create a `arm64` image with prefix of `ghcr.io` and the tag as `nightly`.
For example, you can use the following command to create a `arm64`
image with the tag `gcr.io/alcionai/corso:latest`, you can run:
```bash
./build/build-container.sh --arch arm64 --prefix ghcr.io --suffix nightly
./build/build.sh image --platforms linux/arm64 --tag gcr.io/alcionai/corso:latest
```
:::info
If you run into any issues with building cross platform images, make
sure to follow the instructions on [Docker
docs](https://docs.docker.com/build/building/multi-platform/) to setup
the build environment for Multi-platform images.
:::