dustin/jenkinsagent/pipeline/head This commit looks goodDetails
There is apparently a bug in *runc* 1.1.0 that prevents privileged
containers from starting:
> docker: Error response from daemon: failed to create shim: OCI runtime
> create failed: runc create failed: unable to start container process:
> unable to apply cgroup configuration: chown
> /sys/fs/cgroup/system.slice/docker-….scope/memory.oom.group: no such
> file or directory: unknown.
A patch is available but has not been integrated into an official
release yet.
Since Git does not track empty directories, the `/var/lib/rsyslog`
directory was not included in the overlay hierarchy, and thus missing
from the final rootfs image. The directory needs to have a file within
it in order to be tracked. Buildroot automatically skips copying files
named `.keep` from the overlay, presumably for this reason.
When the DHCP lease is renewed, if the specified log servers are the
same as the previous lease, there is no reason to regenerate the
*rsyslog* configuration file and restart the service.
Jenkins agents need to have a pre-built cache of SSH public keys for any
host jobs need to access. This file is typically bind-mounted into
jobs' containers.
For the older Fedora-based agent nodes, the `ssh_known_hosts` file is
populated by Ansible. That mechanism will not work for agent nodes
using the immutable root filesystem created by this project, so we need
an alternative. To that end, the `fetch-ssh-knownhosts` service
downloads the keys from another machine using HTTPS when the system
boots up and then periodically while it is running.
Using the Busybox NTP client, we can set the system clock in early
userspace. This will ensure the system clock is correct before
anything, even systemd, is started from the real rootfs.
Obviously, this requires the DHCP server to specify NTP server addresses
in the client lease.
If systemd detects that the system clock is not correct, it sets it to a
"known good" time based on the release date of its version. Since this
can still be weeks or months in the past, it will use the timestamp of
the `/usr/lib/clock-epoch` file, if it exists, as the reference time.
To get our system clock closer to the correct time earlier in the boot
process, we update the `clock-epoch` file's timestamp during build.
This will cause the clock to be set to approximately the build time of
the root filesystem image. Presumably, the rootfs image is built
relatively frequently, so the time should be a lot closer to correct
than the systemd release date.
Docker and Jenkins need more storage than is available in the *tmpfs*
filesystem mounted at `/var`. We can use NBD to provide additional
storage volumes for these paths.
The `nbd@.service` systemd unit, which is included in the *nbd* package
but not actually installed by it, starts an NBD client, using the
configuration in `/etc/nbdtab` for the specified block device. To avoid
hard-coding the NBD server name here, the file included in the rootfs
image is actually a template. This template is rendered at boot time,
using the same server name specified on the kernel command line for the
rootfs device. Obviously, this implies that the same server has to host
both the rootfs image and the persistent storage, but that is not likely
to be a problem for this project.
To allow multiple devices to use the same rootfs image but separate
persistent storage devices, the `nbdtab` template can include a
placeholder for the device's serial number. This only works for
Raspberry Pi devices, of course, but the concept could be extended to
other device types by adding more placeholders for other
machine-specific properties (e.g. DMI UUID, etc.)
Since `/var/lib/jenkins` is the home directory for the *jenkins* user,
and thus the location of its `~/.ssh/authorized_keys` file, we have to
make sure the *fetch-sshkeys@jenkins.service* unit does not start until
the persistent storage volume is mounted. We also need a service unit
that ensures the permissions of the mount point are correct,
particularly for the first boot when the filesystem is initially
created.
Similarly, we have to ensure that the Docker daemon does not start until
`/var/lib/docker` is mounted. Adding a `RequiresMountsFor=` property to
the *docker.service* unit generates the necessary dependencies to
accomplish this.
Using a systemd generator to automatically enable
*fetch-sshkeys@.service* unit instances allows multiple users' keys to
be specified on the kernel command line, without explicitly enabling
the corresponding units.
Since `/var` is a tmpfs filesystem, having the Journal stored there is
just a waste of space; `systemd-journald` already writes the journal to
`/run/log/journal`.
*dhcpcd* does not set the system host name to the value provided in the
DHCP lease unless the current hostname matches its compiled-in default
value (which is `(none)`) or is *localhost*. I'm not sure how to change
the former, but we can easily set the default to the latter to enable
dynamic hostname configuration via DHCP.
Instead of parsing the kernel command line to find the destination(s)
where syslog messages should be forwarded, we can use a *dhcpcd* hook to
get them from the DHCP lease.
By default, *dhcpcd* generates a "complete" `chrony.conf` file using the
NTP servers specified in the DHCP lease, and then restarts the Chrony
daemon. This is unnecessary, as *chrony* can reload its list of
sources without restarting, and also problematic in our case. For one
thing, *dhcpcd* tries to write to `/etc/chrony.conf` directly, which is
obviously impossible when the root filesystem is immutable.
Fortunately, it is flexible enough to be configured to leave
`/etc/chrony.conf` alone and generate a different file and then tell
*chrony* to reread it.
The *chrony-wait.service* unit introduces a delay into the system
startup that allows follow-up units to wait until the system time is
confirmed correct. The unit file itself was derived from the one
included in the *chrony* package for Fedora.
Instead of manually managing symlinks to enable units, we can define a
system preset that lists which units we want disabled. Units that have
in `Install` section are automatically enabled.
Dustin