*serial0.pyrocufflink.blue* has been replaced by
*serial1.pyrocufflink.blue*. The latter runs Fedora CoreOS and is
managed by the CUE-based configuration policy in *cfg.git*.
Recent versions of NUT have a *nut.target* unit that collects all of the
NUT-related services. Enabling any of the services individually does
effectively nothing, as it only adds the service as a `Wants` dependency
for *nut.target*, and that unit already has dependencies for all of
them. Thus, in order for the service to start at boot, *nut.target* has
to be enabled instead.
In situations where only *nut-monitor* should be enabled, enabling
*nut.target* is inappropriate, since that enables *nut-driver* and
*nut-server* as well. It's not clear why upstream made this change (it
was part of a [HUGE pull request][0]), but restoring the desired
behavior is easy enough by clearing the dependencies from *nut.target*.
Services that we want to start automatically can still be enabled
individually, and will start as long as *nut.target* is enabled.
[0]: https://github.com/networkupstools/nut/pull/330
The automatic transfer switch does not seem to work reliably when both
UPS sources are running on battery. This means all systems lose power
after the first UPS battery dies, even though the second UPS is still
online. To minimize the risk of data loss, at least until I figure out
what's wrong, I want both VM hosts to shut down as soon as the first UPS
signals that its battery is low.
Increasing the delay after starting the Kubernetes cluster to hopefully
allow things to "settle down" enough that starting services on follow up
VMs doesn't time out.
The Synapse server can sometimes take a very long time to start.
Increasing the start timeout should keep it from failing to come up when
the machine is under load.
Start Kubernetes earlier. Start Synapse later (it takes a long time to
start up and often times out when the VM hosts are under heavy load).
Start SMTP relay later as it's not really needed.
The UniFi controller service can sometimes take a really long time to
start up. This most frequently happens after a full outage, when the VM
hosts are very busy bringing everything up.
The network device for the test/*pyrocufflink.red* network is named
`br1`. This needs to match in the systemd-networkd configuration or
libvirt will not be able to attach virtual machines to the bridge.
The `vm-autostart` script fails with `bad system call` errors when
trying to start libvirt domains. Removing the system call filters works
around this. Ideally, we should figure out exactly which system call is
being rejected and allow it, but that's rather difficult to do and
probably not really worth the effort in this case.
The MinIO service often fails to start from a cold boot. Delaying
starting the service until the network is online, plus increasing the
startup timeout, should help with this. If not, enabling auto restart
will let systemd try to start the service again if it still fails to
come up on time.
`upsmon` is the component of [NUT] that monitors (local or remote) UPS
devices and reacts to changes in their state. Notably, it is
responsible for powering down the system when there is insufficient
power to the system.
Adding the `always` tag to the tasks in `dyngroups.yml` ensures that
hosts will get added to the appropriate groups dynamically, even when
running a subset of tasks by targeting specific tags. Ansible will
always run tasks with this tag when a tag selection is passed to
`ansible-playbook`.
The Metrics Pi has bit the dust. The NVMe disk has never been
particularly reliable, but now it's gotten to the point where it's a
real issue. The Pi needs rebooted at least once a day.
I've moved the Victoria Metrics/Grafana ecosystem to Kubernetes.
Since *file0.pyrocufflink.blue* now hosts a couple of VirtualHosts,
accessing its HTTP server by the *files.pyrocufflink.blue* alias no
longer works, as Apache routes unknown hostnames to the first
VirtualHost, rather than the global configuration. To resolve this, we
must set `ServerName` to the alias.
I've moved the Dark Chest of Wonders website to run in a container on
Kubernetes. This will keep it from breaking every time the OS is
updated on the web server, when the version of Python in Fedora changes.
Recent(-ish) versions of Fedora have a drop-in configuration directory
for `sshd`. This allows applications, etc. to define certain settings
for the SSH server, without having to manage the entire server
configuration. For Gitea specifically, we only need to set a few
settings for the *gitea* user, leaving the remaining settings alone.
This commit does not include any migration to undo the settings that
were originally set, but that should be as simple as `mv
/etc/ssh/sshd_config.rpmnew /etc/ssh/sshd_config && systemctl reload
sshd`.
The *ssh-host-certs* role, which is now applied as part of the
`base.yml` playbook and therefore applies to all managed nodes, is
responsible for installing the *sshca-cli* package and using it to
request signed SSH host certificates. The *sshca-cli-systemd*
sub-package includes systemd units that automate the process of
requesting and renewing host certificates. These units need to be
enabled and provided the URL of the SSHCA service. Additionally, the
SSH daemon needs to be configured to load the host certificates.
The *dch* repository, hosted on *file0.pyrocufflink.blue* and managed by
the *repohost* Ansible role, is where I plan to host RPM packages for
internal use (e.g. *sshca-cli*, *dch-selinux*, etc.). The *dch-yum*
role configures Yum/dnf to use this repository. Roles that need to
install a package from here will list this role as a dependency.
So it turns out Gitea's RPM package repository feature is less than
stellar. Since each organization/user can only have a single
repository, separating packages by OS would be extremely cumbersome.
Presumably, the feature was designed for projects that only build a
single PRM for each version, but most of my packages need multiple
builds, as they tend to link to system libraries. Further, only the
repository owner can publish to user-scoped repositories, so e.g.
Jenkins cannot publish anything to a repository under my *dustin*
account. This means I would ultimately have to create an Organization
for every OS/version I need to support, and make Jenkins a member of it.
That sounds tedious and annoying, so I decided against using that
feature for internal packages.
Instead, I decided to return to the old ways, publishing packages with
`rsync` and serving them with Apache. It's fairly straightforward to
set this up: just need a directory with the appropriate permissions for
users to upload packages, and configure Apache to serve from it.
One advantage Gitea's feature had over a plain directory is its
automatic management of repository metadata. Publishers only have to
upload the RPMs they want to serve, and Gitea handles generating the
index, database, etc. files necessary to make the packages available to
Yum/dnf. With a plain file host, the publisher would need to use
`createrepo` to generate the repository metadata and upload that as
well. For repositories with multiple packages, the publisher would need
a copy of every RPM file locally in order for them to be included in the
repository metadata. This, too, seems like it would be too much trouble
to be tenable, so I created a simple automatic metadata manager for the
file-based repo host. Using `inotifywatch`, the `repohost-createrepo`
script watches for file modifications in the repository base directory.
Whenever a file is added or changed, the directory containing it is
added to a queue. Every thirty seconds, the queue is processed; for
each unique directory in the queue, repository metadata are generated.
This implementation combines the flexibility of a plain file host,
supporting an effectively unlimited number of repositories with
fully-configurable permissions, and the ease of publishing of a simple
file upload.
The `net cache flush` command does not seem to always work to clear the
identity mapping cache used by winbind. Explicitly moving the file
does, though.
On a new DC, the `idmap.ldb` file does not yet exist the first time
`sysvolsync` runs. This causes a syntax error in the condition that
checks the modification timestamp of the file.
The *sudo* role should not be applied by the `samba-dc.yml` playbook.
It's not relevant to deploying Samba, and will just get applied by the
`domain-controller.yml` playbook later, anway. Further, it fails when
a new domain controller is first provisioned, because at this step,
the system is not yet configured to resolve user IDs via winbind;
rather than add users to groups, it tries to create them.
Forcing the PDC lookup to use localhost as the DNS server does not work
when first adding a new domain controller, as the `sysvolsync` script
runs before Samba starts. There isn't much advantage to using the local
DNS server over the system-defined server anyway.
The `winbind offline login` setting seems to cause issues when one of
the domain controllers is offline. Rather than try the other DC,
winbind seems to just "give up" and return NT_STATUS_NO_SUCH_USER for
all authentication requests until the offline cache is flushed. There's
not really any reason to use this setting on servers anyway, since they
are always connected to the LAN, as opposed to laptops that may
occasionally disconnect. Let's disable this option in the hopes that it
makes logins more resilient to DC downtime. After all, there's not much
point in having multiple DCs if they all have to be available in order
to log in.
AWS is going to begin charging extra for routable IPv4 addresses soon.
There's really no point in having a relay in the cloud anymore anyway,
since a) all outbound messages are sent via the local relay and b) no
messages are sent to anyone except me.
The *authconfig* package has been gone from Fedora since ages. There's
no reason to have this no-op step any more, especially since it has the
side-effect of making a network request to refresh the dnf cache.
I've added a new Kubernetes worker node,
*k8s-aarch64-n1.pyrocufflink.blue*. This machine is a Raspberry Pi CM4
mounted on a Waveshare CM4-IO-Base A and clipped onto the DIN rail.
It's got 8 GB of RAM and 32 GB of eMMC storage. I intend to use it to
build container images locally, instead of bringing up cloud instances.
Zincati is the automatic update manager on Fedora CoreOS. It exposes
Prometheus metrics for host/update statistics, which are useful to track
the progress of automatic updates and identify update issues.
Zinciti actually exposes its metrics via a Unix socket on the
filesystem. Another process, [local_exporter], is required to expose
the metrics from this socket via HTTP so Prometheus can scrape them.
[local_exporter]: https://github.com/lucab/local_exporter
*collectd* is now running on *k8s-aarch64-n0.pyrocufflink.blue*,
exposing system metrics. As it is not a member of the AD domain, it has
to be explicitly listed in the `scrape_collectd_extra_targets` variable.
*nvr1.pyrocufflink.blue* has been migrated to Fedora CoreOS. As such,
it is no longer managed by Ansible; its configuration is done via
Butane/Ignition. It is no longer a member of the Active Directory
domain, but it does still run *collectd* and export Prometheus metrics.
Dustin