Buildah 1.38 and 1.39 have a bug that prevents them being able to build
containers in a Kubernetes pod. Whenever I try, I get:
> error stat()ing intended root directory
> "/var/tmp/buildah3537271510/mnt/rootfs": no such file or directory
I have not identified the root cause yet, but reverting to 1.37 works
around the issue for now.
For aarch64 builds that cannot run on Raspberry Pi nodes, the pod must
tolerate the `du5t1n.me/jenkins` taint, which is applied to ephemeral
nodes in EC2. Without this toleration, the pod can never be scheduled,
as there are no aarch64 nodes besides Rasberry Pis that do not have that
taint.
Jobs that need to compile code, as opposed to simply installing
prebuilt packages and/or copying in files, probably do not want to run
on a Raspberry Pi. To allow such jobs to opt out of running on a Pi and
wait for the Cluster Autoscaler to bring up an aarch64 node in AWS, the
`buildContainerImage2` function now takes an optional `pi` argument.
The argument defaults to `true`, so existing jobs that expect to run on
a Pi will continue to do so.
In order to keep "big" applications that just happen to have arm64
container images, like Argo CD and Jenkins, from running on Raspberry
Pis instead of VMs, I added a `du5t1n.me/machine=raspberrypi:NoExecute`
taint to them. This requires any workload that should be allowed to run
on a Raspberry Pi to explicitly indicate as much. Since the point of
adding *k8s-aarch64-n1.pyrocufflink.blue* to the cluster was to be able
to build container images, the pod template for container image jobs
needs to include this toleration.
The `buildContainerImage2` function is an improvement on the existing
`buildContainerImage` function, with a couple of enhancements.
Primarily, it supports building images for multiple architectures. For
each listed architecture, a pod is launched on a matching worker node to
build the image natively. The image is exported to an OCI archive and
"stashed" in the Jenkins workspace. After images for all architectures
are built, another pod is launched and all of the stashed archives are
copied there and assembled into a manifest list. Finally, the manifest
list is published to the OCI repository as usual, creating tags for the
Git branch and build number.
In addition to adding support for multiarch images, the
`buildContainerImage2` performs image builds in *unprivileged* pods.
Whereas the old function performed builds in a rootful container, the
new one configures requests pods with unique user namespaces. The build
runs as *root* in the container, but that user is mapped to an arbitrary
unprivileged user on the host.
Dustin