Instead of a top-level boolean that says whether or not the build
can/should run on a Raspberry Pi, we'll expose a `resources` argument
that allows jobs to specify their compute requirements. If they're not
specified, then the job can run anywhere. Otherwise, if the
requirements are too large for a Pi, then the autoscaler will launch a
cloud worker to handle the build.
Buildah 1.38 and 1.39 have a bug that prevents them being able to build
containers in a Kubernetes pod. Whenever I try, I get:
> error stat()ing intended root directory
> "/var/tmp/buildah3537271510/mnt/rootfs": no such file or directory
I have not identified the root cause yet, but reverting to 1.37 works
around the issue for now.
For aarch64 builds that cannot run on Raspberry Pi nodes, the pod must
tolerate the `du5t1n.me/jenkins` taint, which is applied to ephemeral
nodes in EC2. Without this toleration, the pod can never be scheduled,
as there are no aarch64 nodes besides Rasberry Pis that do not have that
taint.
Jobs that need to compile code, as opposed to simply installing
prebuilt packages and/or copying in files, probably do not want to run
on a Raspberry Pi. To allow such jobs to opt out of running on a Pi and
wait for the Cluster Autoscaler to bring up an aarch64 node in AWS, the
`buildContainerImage2` function now takes an optional `pi` argument.
The argument defaults to `true`, so existing jobs that expect to run on
a Pi will continue to do so.
Passing the `defaultBranch` argument to `buildContainerImage2()` allows
setting the `latest` for projects where the default branch is not
*main*. For example:
```groovy
buildContainerImage2(defaultBranch: 'master')
```
This will update the `latest` tag whenever a new build is created on the
*master* branch.
The `buildContainerImage2` function now accepts an optional `schedule`
argument. If specified, the value will be used as the cron expression
for triggering a build periodically.
In order to keep "big" applications that just happen to have arm64
container images, like Argo CD and Jenkins, from running on Raspberry
Pis instead of VMs, I added a `du5t1n.me/machine=raspberrypi:NoExecute`
taint to them. This requires any workload that should be allowed to run
on a Raspberry Pi to explicitly indicate as much. Since the point of
adding *k8s-aarch64-n1.pyrocufflink.blue* to the cluster was to be able
to build container images, the pod template for container image jobs
needs to include this toleration.
If the image being built is only for a single architecture, the step to
unstash the OCI archive file was missing. This caused the build to fail
at the Push step, since there was noting to push.
If the provided container image name includes a `/` character, the build
will fail when copying the OCI image to a tarball:
> + buildah push git.pyrocufflink.net/containerimages/build/selinux:dev-ci oci-archive:/home/jenkins/agent/workspace/ainerImages_build.selinux_dev_ci/build/selinux-amd64.tar
> Error: lstat /home/jenkins/agent/workspace/ainerImages_build.selinux_dev_ci/build: no such file or directory
To resolve this, we need to escape the image name when constructing the
path to the tar file.
The `buildContainerImage2` function is an improvement on the existing
`buildContainerImage` function, with a couple of enhancements.
Primarily, it supports building images for multiple architectures. For
each listed architecture, a pod is launched on a matching worker node to
build the image natively. The image is exported to an OCI archive and
"stashed" in the Jenkins workspace. After images for all architectures
are built, another pod is launched and all of the stashed archives are
copied there and assembled into a manifest list. Finally, the manifest
list is published to the OCI repository as usual, creating tags for the
Git branch and build number.
In addition to adding support for multiarch images, the
`buildContainerImage2` performs image builds in *unprivileged* pods.
Whereas the old function performed builds in a rootful container, the
new one configures requests pods with unique user namespaces. The build
runs as *root* in the container, but that user is mapped to an arbitrary
unprivileged user on the host.
If the `buildContainerImage` method is called without passing any
arguments at all, the `args` object will be `null`. Attempting to
access the properties of it in that case raises a
`NullPointerException`. Fortunately, Groovy has a `?` operator that
returns `null` when the named object is `null` and avoids accessing its
properties.
The `buildContainerImage` method now supports an optional `arch` keyword
argument. This argument can be used to select the architecture of the
node running the pod building the container image. If unspecified, it
defaults to `amd64`.
The `registry`, `project`, `name`, and `tag` values can now be passed as
keyword arguments to the `buildContainerImage` function. This will
allow jobs to override the default values if necessary.
Dustin