Jenkins agents need to have a pre-built cache of SSH public keys for any
host jobs need to access. This file is typically bind-mounted into
jobs' containers.
For the older Fedora-based agent nodes, the `ssh_known_hosts` file is
populated by Ansible. That mechanism will not work for agent nodes
using the immutable root filesystem created by this project, so we need
an alternative. To that end, the `fetch-ssh-knownhosts` service
downloads the keys from another machine using HTTPS when the system
boots up and then periodically while it is running.
Docker and Jenkins need more storage than is available in the *tmpfs*
filesystem mounted at `/var`. We can use NBD to provide additional
storage volumes for these paths.
The `nbd@.service` systemd unit, which is included in the *nbd* package
but not actually installed by it, starts an NBD client, using the
configuration in `/etc/nbdtab` for the specified block device. To avoid
hard-coding the NBD server name here, the file included in the rootfs
image is actually a template. This template is rendered at boot time,
using the same server name specified on the kernel command line for the
rootfs device. Obviously, this implies that the same server has to host
both the rootfs image and the persistent storage, but that is not likely
to be a problem for this project.
To allow multiple devices to use the same rootfs image but separate
persistent storage devices, the `nbdtab` template can include a
placeholder for the device's serial number. This only works for
Raspberry Pi devices, of course, but the concept could be extended to
other device types by adding more placeholders for other
machine-specific properties (e.g. DMI UUID, etc.)
Since `/var/lib/jenkins` is the home directory for the *jenkins* user,
and thus the location of its `~/.ssh/authorized_keys` file, we have to
make sure the *fetch-sshkeys@jenkins.service* unit does not start until
the persistent storage volume is mounted. We also need a service unit
that ensures the permissions of the mount point are correct,
particularly for the first boot when the filesystem is initially
created.
Similarly, we have to ensure that the Docker daemon does not start until
`/var/lib/docker` is mounted. Adding a `RequiresMountsFor=` property to
the *docker.service* unit generates the necessary dependencies to
accomplish this.
Using a systemd generator to automatically enable
*fetch-sshkeys@.service* unit instances allows multiple users' keys to
be specified on the kernel command line, without explicitly enabling
the corresponding units.
Instead of parsing the kernel command line to find the destination(s)
where syslog messages should be forwarded, we can use a *dhcpcd* hook to
get them from the DHCP lease.
By default, *dhcpcd* generates a "complete" `chrony.conf` file using the
NTP servers specified in the DHCP lease, and then restarts the Chrony
daemon. This is unnecessary, as *chrony* can reload its list of
sources without restarting, and also problematic in our case. For one
thing, *dhcpcd* tries to write to `/etc/chrony.conf` directly, which is
obviously impossible when the root filesystem is immutable.
Fortunately, it is flexible enough to be configured to leave
`/etc/chrony.conf` alone and generate a different file and then tell
*chrony* to reread it.
The *chrony-wait.service* unit introduces a delay into the system
startup that allows follow-up units to wait until the system time is
confirmed correct. The unit file itself was derived from the one
included in the *chrony* package for Fedora.
Instead of manually managing symlinks to enable units, we can define a
system preset that lists which units we want disabled. Units that have
in `Install` section are automatically enabled.
Dustin