7 Commits

Author SHA1 Message Date
62e61522e5 ci: post: Trigger correct job, do not wait
Some checks failed
Aimee OS/container-images/pipeline/head There was a failure building this commit
The next job in the pipeline is _rust-cross/main_, not
_cross-rust/main_.

Also, we don't need to wait for follow-up jobs.  Otherwise, we end up
with the whole pipeline blocking the _base_ job, etc.
2025-01-03 15:49:56 -06:00
ad77a3a035 ci: Trigger cross-rust build on success
Some checks failed
Aimee OS/container-images/pipeline/head There was a failure building this commit
When the "cross" image finishes building, Jenkins will start building a
new "cross-rust" image on top of it.
2024-12-22 09:16:29 -06:00
7946b8c084 Do not cross-compile Rust
This will happen in a separate layer.
2024-12-22 09:15:47 -06:00
825e1c17b1 ci: Mount ephemeral volume for container storage
Some checks failed
Aimee OS/container-images/pipeline/head There was a failure building this commit
Without a specific volume mount, the Buildah working container will be
stored in the Jenkins pod container filesystem.  This not only affects
performance, but limits the size of the working container filesystem and
the final image, as the worker nodes do not have very much space for
container filesystems.  Thus, we need to mount an ephemeral Longhorn
volume in the job pod to provide more space to Buildah.

Using a read-only root filesystem helps ensure that no temporary data
get written to container storage.  This of course breaks the work-around
we had in place for overriding the broken default `storage.conf` in the
_buildah_ image, so we mount a _tmpfs_ filesystem at `/home/build` as a
different work-around.
2024-12-21 15:09:07 -06:00
79a2645c17 Correctly configure USE/env vars for Rust
Although `LLVM_TARGETS` is USE-forced for _llvm-core/llvm_, it is NOT
forced for _dev-lang/rust_.  The latter needs the target architecture
set in order to build the cross-compiled Rust standard library.
2024-12-21 15:09:07 -06:00
48fd2abbda build: Emerge gettext with USE=-nls first
This is necessary to break a dependency cycle between
_sys-devel/gettext_ and _sys-devel/m4[nls]_.
2024-12-21 15:09:07 -06:00
359e5f3c3b split-repo: Populate cross branch
In order to create multiple Jenkins build pipelines for this repository,
I'm splitting it up with branches.  There will be a branch for each
image:

* base/main: The base image
* cross/main: The image with a cross-compiler toolchain
* build/main: The final build image

Note _/main_ in each name.  The intent here is to be able to have
different "sub-branches" for each image, so e.g. there might be a
_base/dev_.

In addition to the branches for each image, there will be branches for
the shared library code as well, named like _lib/main_.  The image
branches will checkout this branch using `git worktree`.
2024-12-21 15:09:07 -06:00
31 changed files with 68 additions and 377 deletions

View File

@@ -1,62 +0,0 @@
# Aimee OS Build Container Images
Prerequisites:
* `buildah`
## Gentoo Stage 3 Image
This image is used to bootstrap the base layer for Aimee OS build images. To
build it, use the `gentoo-stage3.sh` build script. This script takes a single
(optional) positional argument: the name of a Gentoo stage 3 tarball to use to
produce the container image. If none is specified, the latest
`stage3-amd64-nomultilib-openrc` tarball is downloaded and used.
```sh
sh image-build/gentoo-stage3.sh
```
## Base Image
The Aimee OS base build image is a minimal environment based on Gentoo. It
includes Portage and a basic build toolchain. Building it requires a Stage 3
image to bootstrap the environment. By default,
_aimee-os.org/gentoo/stage3-amd64-nomultilib-openrc:latest_ is used, but an
alternative image can be specified using the `-b`/`--base` argument to the
`base.sh` script.
```sh
sh image-build/base.sh
```
## Crossdev Image
This image includes a cross-compilation toolchain, generated by _crossdev_. It
is built using the `cross.sh` script. This script takes a `-t`/`--target`
argument, which specifies the toolchain to generate. The default is
`aarch64-unknown-linux-gnu`. This image is layered on top of the base image,
which can be specified using the `-b`/`--base` argument. By default,
_aimee-os.org/build/base:latest_ is used.
```sh
sh image-build/cross.sh
```
## Build Image
The top-level image is based on the Crossdev image, and includes additional
tools for building an Aimee OS project. It is built using the `build.sh` script.
As with the Crossdev image build script, this script accepts `-t`/`--target`
and `-b`/`--build` arguments, which specify the cross-compilation toolchain and
base image later, respectively.
```sh
sh image-build/build.sh
```
This image is "squashed," meaning it consists of only a single layer. The Base
and Crossdev image layers are merged into the final layer when the image is
built. This reduces the final size of the image.

View File

@@ -1,7 +1,7 @@
#!/bin/sh
SELF=$(readlink -f "$0")
SRCDIR=${SELF%/*/*}
SRCDIR=${SELF%/*}
. "${SRCDIR}"/lib/common.sh
target=aarch64-unknown-linux-gnu
@@ -28,5 +28,5 @@ set -e
cid=$(buildah from "${base}")
buildah_run_script "${cid}" "${SRCDIR}"/lib/sync.sh
buildah_run_script "${cid}" "${SRCDIR}"/lib/crossdev.sh "${target}"
buildah_run_script "${cid}" "${SRCDIR}"/crossdev.sh "${target}"
buildah commit --rm "${cid}" aimee-os.org/build/cross-"${target}"

31
ci/Jenkinsfile vendored
View File

@@ -8,27 +8,19 @@ pipeline {
}
stages {
stage('Gentoo Stage 3') {
stage('Prepare') {
steps {
sh 'sh -e image-build/gentoo-stage3.sh'
}
}
stage('Base') {
steps {
sh 'sh -e image-build/base.sh'
}
}
stage('Cross') {
steps {
sh 'sh -e image-build/cross.sh'
container('jnlp') {
// TODO checkout lib/ branch based on $BRANCH_NAME
sh 'git fetch origin lib/main:lib/main'
sh 'git worktree add lib lib/main'
}
}
}
stage('Build') {
steps {
sh 'sh -e image-build/build.sh'
sh 'sh -e build.sh --base git.pyrocufflink.net/aimeeos/build/base'
}
}
@@ -49,9 +41,7 @@ pipeline {
git.pyrocufflink.net
"""
}
sh 'buildah push aimee-os.org/build/base git.pyrocufflink.net/aimeeos/build/base'
sh 'buildah push aimee-os.org/build/cross-aarch64-unknown-linux-gnu git.pyrocufflink.net/aimeeos/build/cross-aarch64-unknown-linux-gnu'
sh 'buildah push aimee-os.org/build/build-aarch64-unknown-linux-gnu git.pyrocufflink.net/aimeeos/build/build-aarch64-unknown-linux-gnu'
}
}
}
@@ -59,6 +49,13 @@ pipeline {
}
post {
success {
build([
job: "${BRANCH_NAME.replace('cross/', 'rust-cross%2F')}",
wait: false,
])
}
failure {
sh 'unshare -Ur --map-auto chown root:root -R tmp log'
dir('tmp/portage') {

View File

@@ -7,12 +7,12 @@ spec:
- -c
- |
trap 'kill $!; exit' TERM
rm -f ~/.config/containers/storage.conf
sleep infinity &
wait
securityContext:
runAsUser: 1000
runAsGroup: 1000
readOnlyRootFilesystem: true
resources:
limits:
github.com/fuse: 1
@@ -21,5 +21,31 @@ spec:
requests:
cpu: 6
memory: 8G
volumeMounts:
- mountPath: /home/build
name: tmp
subPath: home
- mountPath: /home/build/.local/share/containers
name: data
subPath: containers
- mountPath: /tmp
name: tmp
subPath: tmp
- mountPath: /var/tmp
name: data
subPath: tmp
tolerations:
- key: du5t1n.me/jenkins
volumes:
- name: data
ephemeral:
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
- name: tmp
emptyDir:
medium: Memory

25
crossdev.sh Executable file
View File

@@ -0,0 +1,25 @@
#!/bin/sh
set -e
target=$1
USE=-nls \
emerge -kvnj sys-devel/gettext
mkdir -p /var/db/repos/crossdev/profiles /var/db/repos/crossdev/metadata
echo crossdev > /var/db/repos/crossdev/profiles/repo_name
echo 'masters = gentoo' > /var/db/repos/crossdev/metadata/layout.conf
chown -R portage:portage /var/db/repos/crossdev
mkdir -p /etc/portage/repos.conf
cat > /etc/portage/repos.conf/crossdev.conf <<EOF
[crossdev]
location = /var/db/repos/crossdev
priority = 10
masters = gentoo
auto-sync = no
EOF
emerge -kvnj --rebuilt-binaries=y sys-devel/crossdev
crossdev --stable -t "$1"

View File

@@ -1,39 +0,0 @@
#!/bin/sh
SELF=$(readlink -f "$0")
SRCDIR=${SELF%/*/*}
. "${SRCDIR}"/lib/common.sh
base=aimee-os.org/gentoo/stage3-amd64-nomultilib-openrc:latest
while [ $# -gt 0 ]; do
case "$1" in
-b|--base)
shift
base="$1"
;;
*)
printf 'Unknown argument: %s\n' "$1" >&2
exit 2
;;
esac
done
set -e
builddir=$(export TMPDIR=${TMPDIR:-${PWD}} && mktemp -d gentoo.XXXXXX)
builddir=$(readlink -f "${builddir}")
work=$(buildah from -v "${builddir}":/mnt/gentoo:rw,Z "${base}")
buildah add "${work}" portage /etc/portage/
buildah_run_script "${work}" "${SRCDIR}"/lib/sync.sh
buildah_run_script "${work}" "${SRCDIR}"/lib/bootstrap.sh
cid=$(buildah from scratch)
buildah copy "${cid}" "${builddir}" /
buildah_run_script "${cid}" "${SRCDIR}"/lib/profile.sh
buildah config --cmd /bin/bash "${cid}"
buildah commit --rm --squash "${cid}" aimee-os.org/build/base
buildah run "${work}" find /mnt/gentoo -mindepth 1 -delete
buildah rm "${work}"
rmdir "${builddir}"

View File

@@ -1,35 +0,0 @@
#!/bin/sh
SELF=$(readlink -f "$0")
SRCDIR=${SELF%/*/*}
. "${SRCDIR}"/lib/common.sh
target=aarch64-unknown-linux-gnu
while [ $# -gt 0 ]; do
case "$1" in
-t|--target)
shift
target="$1"
;;
-b|--base)
shift
base="$1"
;;
*)
printf 'Unknown argument: %s\n' "$1" >&2
exit 2
;;
esac
done
if [ -z "${base-}" ]; then
base=aimee-os.org/build/cross-"${target}"
fi
set -e
cid=$(buildah from "${base}")
buildah add "${cid}" portage /etc/portage
buildah_run_script "${cid}" "${SRCDIR}"/lib/sync.sh
buildah_run_script "${cid}" "${SRCDIR}"/lib/tools.sh
buildah commit --rm --squash "${cid}" "aimee-os.org/build/build-${target}"

View File

@@ -1,22 +0,0 @@
#!/bin/sh
SELF=$(readlink -f "$0")
SRCDIR=${SELF%/*/*}
tarball=$1
set -e
if [ -z "${tarball}" ]; then
tarball=$(sh "${SRCDIR}"/tools/fetch-stage3.sh "$@")
fi
stage3=${tarball%.tar.*}
version=${stage3##*-}
stage3=${stage3%-*}
cid=$(buildah from scratch)
buildah add "${cid}" "${tarball}"
buildah config --cmd /bin/bash "${cid}"
buildah commit --rm --squash "${cid}" "aimee-os.org/gentoo/${stage3}:${version}"
buildah tag "aimee-os.org/gentoo/${stage3}:${version}" \
"aimee-os.org/gentoo/${stage3}:latest"

View File

@@ -1,29 +0,0 @@
#!/bin/sh
set -e
mkdir -p /mnt/gentoo
USE=build \
emerge -kvnuUDj --rebuilt-binaries=y --root=/mnt/gentoo sys-apps/baselayout
# Portage itself only requires Python and Bash. Actually building
# packages requires a lot more. These dependencies are implicit because
# they are listed in @system. We do not want everything from @system,
# though, so we have to explicitly install several packages.
#
# At some point, we may end up with an Aimee OS profile for Portage.
# If/when that happens, we can override @system to remove the stuff we
# don't want, like virtual/service-manager and virtual/openssh.
xargs emerge -kvnuUDj --rebuilt-binaries=y --root=/mnt/gentoo <<EOF
app-arch/gzip
dev-build/make
net-misc/wget
sys-apps/diffutils
sys-apps/gawk
sys-apps/grep
sys-apps/portage
sys-devel/binutils
sys-devel/gcc
sys-devel/patch
EOF

View File

@@ -1,21 +0,0 @@
#!/bin/sh
buildah_run_script() {
local cid script
cid=$1
shift
script=$(readlink -f "$1")
shift
mkdir -p binpkgs log tmp
buildah run \
--mount type=cache,target=/var/db/repos/gentoo \
--mount type=cache,target=/var/cache \
--mount type=bind,src="${PWD}"/log,target=/var/log,rw,z \
--mount type=bind,src="${PWD}"/tmp,target=/var/tmp,rw,z \
--mount type=bind,src="${PWD}"/binpkgs,target=/var/cache/binpkgs,rw,z \
--mount type=bind,src="${script}",target=/run/script,ro,z \
--tty=false \
-e FEATURES='-ipc-sandbox -network-sandbox -pid-sandbox' \
"${cid}" \
/run/script "$@"
}

View File

@@ -1,52 +0,0 @@
#!/bin/sh
set -e
target=$1
mkdir -p /var/db/repos/crossdev/profiles /var/db/repos/crossdev/metadata
echo crossdev > /var/db/repos/crossdev/profiles/repo_name
echo 'masters = gentoo' > /var/db/repos/crossdev/metadata/layout.conf
chown -R portage:portage /var/db/repos/crossdev
mkdir -p /etc/portage/repos.conf
cat > /etc/portage/repos.conf/crossdev.conf <<EOF
[crossdev]
location = /var/db/repos/crossdev
priority = 10
masters = gentoo
auto-sync = no
EOF
emerge -kvnj --rebuilt-binaries=y sys-devel/crossdev
crossdev --stable -t "$1"
ln -s /var/db/repos/gentoo/sys-devel/rust-std /var/db/repos/crossdev/cross-${target}/
mkdir -p
/etc/portage/env/dev-lang \
/etc/portage/package.accept_keywords \
/etc/portage/package.use \
&& :
printf 'cross-%s/rust-std **\n' "${target}" \
> /etc/portage/package.accept-keywords/rust-cross
echo 'dev-lang/rust rust-src' \
> /etc/portage/package.use/rust-src
case ${target%%-*} in
aarch64)
llvm_target=AArch64
;;
arm*)
llvm_target=ARM
;;
*)
printf 'Unknown LLVM target: %s' "${target%%-*}"
exit 1
;;
esac
printf '( %s:%s:%s )\n' \
"${llvm_target}" \
"${target}" \
"${target}" \
>> /etc/portage/env/dev-lang/rust
emerge -vbknuUj cross-${target}/rust-std

View File

@@ -1,11 +0,0 @@
#!/bin/sh
arch=$(uname -m)
case ${arch} in
x86_64) arch=amd64 ;;
aarch64) arch=arm64 ;;
esac
mkdir -p /etc/portage
eselect profile set default/linux/"${arch}"/23.0/no-multilib

View File

@@ -1,8 +0,0 @@
#!/bin/sh
if [ ! -e /var/db/repos/gentoo/metadata ]; then
emerge-webrsync
fi
if [ "$(find /var/db/repos/gentoo/metadata -newermt '-24 hours' | wc -l)" -eq 0 ]; then
emaint sync
fi

View File

@@ -1,20 +0,0 @@
#!/bin/sh
xargs -r emerge -vnuUj --rebuilt-binaries=y <<EOF
sys-apps/shadow
EOF
xargs -r emerge -vnuUj --rebuilt-binaries=y <<EOF
app-emulation/qemu
sec-policy/selinux-base
sys-apps/policycoreutils
sys-apps/kmod
sys-apps/systemd
sys-boot/grub
sys-fs/btrfs-progs
sys-fs/dosfstools
sys-fs/genimage
sys-fs/mtools
sys-fs/squashfs-tools
tini
EOF

View File

@@ -1 +0,0 @@
FEATURES=-buildpkg

View File

@@ -1 +0,0 @@
ACCEPT_KEYWORDS="${ARCH}"

View File

@@ -1,2 +0,0 @@
FEATURES="${FEATURES} binpkg-multi-instance buildpkg"
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --usepkg --binpkg-respect-use=y"

View File

@@ -1 +0,0 @@
USE="${USE} -nls"

View File

@@ -1,2 +0,0 @@
EMERGE_DEFAULT_OPTS="${EMERGE_DEFAULT_OPTS} --quiet-build=y --quiet-fail=y"
FEATURES="${FEATURES} -news"

View File

@@ -1 +0,0 @@
sys-fs/genimage ~amd64

View File

@@ -1 +0,0 @@
sys-kernel/raspberrypi-sources **

View File

@@ -1,6 +0,0 @@
# vim: set ft=gentoo-package-use :
# Dustin C. Hatch <dustin@hatch.name> (09 Feb 2023)
# Do not build binary packages for kernel sources
sys-kernel/gentoo-sources nobuildpkg
sys-kernel/raspberrypi-sources nobuildpkg

View File

@@ -1 +0,0 @@
sys-fs/btrfs-progs -man

View File

@@ -1 +0,0 @@
dev-vcs/git -perl

View File

@@ -1 +0,0 @@
sys-boot/grub -* GRUB_PLATFORMS: -*

View File

@@ -1,2 +0,0 @@
sys-kernel/gentoo-sources symlink
sys-kernel/raspberrypi-sources symlink

View File

@@ -1,5 +0,0 @@
# vim: set ft=gentoo-package-use :
# Dustin C. Hatch <dustin@hatch.name> (11 Dec 2024)
# Build a minimal QEMU for chrooting into the target environment
app-emulation/qemu -* QEMU_USER_TARGETS: aarch64 PYTHON_TARGETS: python3_12

View File

@@ -1,2 +0,0 @@
sys-libs/libselinux python
sys-process/audit python

View File

@@ -1 +0,0 @@
sys-fs/squashfs-tools zstd

View File

@@ -1,6 +0,0 @@
sys-apps/dbus systemd
sys-apps/systemd -* PYTHON_SINGLE_TARGET: python3_12
sec-policy/selinux-base systemd
sec-policy/selinux-base-policy systemd
virtual/tmpfiles systemd
virtual/udev systemd

View File

@@ -1,24 +0,0 @@
#!/bin/sh
: "${GENTOO_MIRROR:=http://mirror.leaseweb.com/gentoo}"
URLPATH=releases/amd64/autobuilds
GENTOO_KEY=13EBBDBEDE7A12775DFDB1BABB572E0E2D182910
GENTOO_KEYSERVER=hkps://keys.gentoo.org
stage3="${1:-amd64-nomultilib-openrc}"
set -e
gpg --keyserver ${GENTOO_KEYSERVER} --recv-keys ${GENTOO_KEY}
curl -fLO "${GENTOO_MIRROR}/${URLPATH}/latest-stage3-${stage3}.txt"
gpg --verify "latest-stage3-${stage3}.txt"
tarball=$(gpg --decrypt "latest-stage3-${stage3}.txt" | awk '$1!="#"{print $1}')
if [ ! -f "${tarball##*/}" ]; then
curl -fLO "${GENTOO_MIRROR}/${URLPATH}/${tarball}"
fi
if [ ! -f "${tarball##*/}.asc" ]; then
curl -fLO "${GENTOO_MIRROR}/${URLPATH}/${tarball}.asc"
fi
gpg --verify "${tarball##*/}.asc"
echo "${tarball##*/}"