*dnf-automatic* is an add-on for `dnf` that performs scheduled,
automatic updates. It works pretty much how I would want it to:
triggered by a systemd timer, sends email reports upon completion, and
only reboots for kernel et al. updates.
In its default configuration, `dnf-automatic.timer` fires every day. I
want machines to update weekly, but I want them to update on different
days (so as to avoid issues if all the machines reboot at once). Thus,
the _dnf-automatic_ role uses a systemd unit extension to change the
schedule. The day-of-the-week is chosen pseudo-randomly based on the
host name of the managed system.
Even with `Network=host`, Podman tries to write to
`/etc/containers/network` for some reason. Fortunately, it doesn't
actually need to, so we can trick it into working by mounting an empty
*tmpfs* filesystem there.
Even with `Network=host`, Podman tries to write to
`/etc/containers/network` for some reason. Fortunately, it doesn't
actually need to, so we can trick it into working by mounting an empty
*tmpfs* filesystem there.
The summer 2024 enrollment form is more complicated than the other
forms on the HLC site, as it integrates directly with Invoice Ninja. As
such, it's handled by a different backend, which runs in Kubernetes.
The BIND server on the firewall is configured to write query logs and
RPZ rewrite logs to files under `/var/log/named`. We can scrape these
logs with Promtail and use the messages for analytics on the DNS-based
firewall, etc.
The UniFi controller can act as a syslog server, receiving log messages
from managed devices and writing them to files in the `logs/remote`
directory under the application data directory. We can scrape these
logs, in addition to the logs created by the UniFi server itself, with
Promtail to get more information about what's happening on the network.
The *promtail* service runs as an unprivileged user by default, which is
fine in most cases (i.e. when scraping only the Journal), but may not
always be sufficient to read logs from other files. Rather than run
Promtail as root in these cases, we can assign it the
CAP_DAC_READ_SEARCH capability, which will allow it to read any file,
but does not grant it any of root's other privileges.
To enable this functionality, the `promtail_dac_read_search` Ansible
variable can be set to `true` for a host or group. This will create a
systemd unit configuration extension that configures the service to have
the CAP_DAC_READ_SEARCH capability in its ambient set.
*unifi1.pyrocufflink.blue* requires a proxy to access Yum repositories
on the Internet, so it has the `proxy` setting configured globally. The
proxy does NOT allow access to internal resources, however. The
internal repository is directly accessible by that machine, so it needs
to be configured thus.
The Squid "cache log" is where it writes general debug and error
messages. It is distinct from the "access log," which is where it
writes the status of every proxy request. We already had the latter
configured to go to syslog by default (so it would be captured in the
journal), but missed the former.
Promtail is the log sending client for Grafana Loki. For traditional
Linux systems, an RPM package is available from upstream, making
installation fairly simple. Configuration is stored in a YAML file, so
again, it's straightforward to configure via Ansible variables. Really,
the only interesting step is adding the _promtail_ user, which is
created by the RPM package, to the _systemd-journal_ group, so that
Promtail can read the systemd journal files.
The internal DNS server for the *pyrocufflink.blue* et al. domains runs
on the firewall now, and is thus no longer managed by Ansible. Dropping
the group variables so the file encrypted with Ansible Vault can go
away.
The *dustinandtabitha.com* website no longer uses *formsubmit* (the time
for RSVP has **long** passed). Removing the configuration so the
file encrypted with Ansible Vault can go away.
A few hosts have `AuthorizedKeysCommand` set in their *sshd(8)*
configuration. This was my first attempt at centrally managing SSH
keys, using a script which fetched a list of keys for each user from an
HTTP server. This worked most of the time, but I didn't take good care
of the HTTP server, so the script would fail frequently. Now that all
hosts trust the SSH user CA, there is no longer any need for this
"feature."
The `TrustedUserCAKeys` setting for *sshd(8)* tells the server to accept
any certificates signed by keys listed in the specified file.
The authenticating username has to match one of the principals listed in
the certificate, of course.
This role is applied to all machines, via the `base.yml` playbook.
Certificates issued by the user CA managed by SSHCA will therefore be
trusted everywhere. This brings us one step closer to eliminating the
dependency on Active Directory/Samba.
Normal users do not need shell access to the file server, and certainly
should not be allowed to e.g. forward ports through it. Using a `Match`
block, we can apply restrictions to users who do not need administrative
functionality. In this case, we restrict everyone who is not a member
of the *Server Admins* group in the PYROCUFFLINK AD domain.
This doesn't actually work, because the shell expands `~` to the value
of the `HOME` environment variable, but `ssh` ignores that variable and
reads from the path from the user's `passwd` entry. Since managed hosts
all have certificates now, and the CA key is included in the global host
key database, individual host keys are not needed anymore anyway.
This machine is _not_ a member of the _pyrocufflink.blue_ AD domain, so
it does not inherit the settings from that group. Also, Jenkins does
not manage it, so only my personal keys are authorized.
Now that servers are configured to use *pam_ssh_agent_auth* for `sudo`
authentication, the Jenkins CI pipelines no longer need to manage the
raw password for the *jenkins* user. A special SSH public key for
Jenkins is listed in `/etc/security/sudo.authorized_keys`, so as long
as a) the corresponding private key is in the SSH agent and b) SSH
agent forwarding is enabled, Ansible will be able to perform privileged
operations without a password.
By default, `sudo` requires users to authenticate with their passwords
before granting them elevated privileges. It can be configured to
allow (some) users access to (some) privileged commands without
prompting for a password (i.e. `NOPASSWD`), however this has a real
security implication. Disabling the password requirement would
effectively grant *any* program root privileges. Prompting for a
password prevents malicious software from running privileged commands
without the user knowing.
Unfortunately, handling `sudo` authentication for Ansible is quite
cumbersome. For interactive use, the `--ask-become-pass`/`-K` argument
is useful, though entering the password for each invocation of
`ansible-playbook` while iterating on configuration policy development
is a bit tedious. For non-interactive use, though, the password of
course needs to be stored somewhere. Encrypting it with Ansible Vault
is one way to protect it, but it still ends up stored on disk somewhere
and needs to be handled carefully.
*pam_ssh_agent_auth* provides an acceptable solution to both issues. It
is better than disabling `sudo` authentication entirely, but a lot more
convenient than dealing with passwords. It uses the calling user's SSH
agent to assert that the user has access to a private key corresponding
to one of the authorized public keys. Using SSH agent forwarding, that
private key can even exist on a remote machine. If the user does not
have a corresponding private key, `sudo` will fall back to normal
password-based authentication.
The security of this solution is highly dependent on the client to store
keys appropriately. FIDO2 keys are supported, though when used with
Ansible, it is quite annoying to have to touch the token for _every
task_ on _every machine_. Thus, I have created new FIDO2 keys for both
my laptop and my desktop that have the `no-touch-required` option
enabled. This means that in order to use `sudo` remotely, I still need
to have my token plugged in to my computer, but I do not have to tap it
every time it's used.
For Jenkins, a hardware token is obviously impossible, but using a
dedicated key stored as a Jenkins credential is probably sufficient.
The [pam_ssh_agent_auth][0] PAM module authenticates users using keys in
their SSH agent. Using SSH agent forwarding, it can even authenticate
users with keys on a remote system. By adding it to the PAM stack for
`sudo`, we can configure the latter to authenticate users without
requiring a password. For servers especially, this is significantly
more secure than configuring `sudo` not to require a password, while
still being almost as convenient.
For this to work, users need to enable SSH agent forwarding on their
clients, and their public keys have to be listed in the
`/etc/security/sudo.authorized_keys` file. Additionally, although the
documentation suggests otherwise, the `SSH_AUTH_SOCK` environment
variable has to be added to the `env_keep` list in *sudoers(5)*.
[0]: https://github.com/jbeverly/pam_ssh_agent_auth
Running `squid -z` as *root* leaves behind temporary files in
`/dev/shm`. When *squid.service* starts squid, in the proper SELinux
domain, it is unable to access these files and crashes. To avoid this,
we mount a private *tmpfs* so no existing files are accessible in the
service's namespace.
Running Squid on the firewall makes sense; it's a sort of layer-7
firewall, after all. There's not much storage on that machine, though
so we don't really want to cache anything. In fact, it's only purpose
is to allow very limited web access for certain applications. All
outbound traffic is blocked, with two exceptions:
* Fedora package repositories (for the UniFi controller server)
* Google Fonts (for Invoice Ninja)
Instead of hard-coding a single cache directory and a set of refresh
patterns, the *squid* role can now have custom cache rules defined with
the `squid_cache_dir` variable (which now takes a list of `cache_dir`
settings) and the `squid_refresh_pattern` variable (which takes a list
of refresh patterns). If neither of these are defined, the default
configuration is used.
This is a breaking change, since `squid_cache_dir` used to refer to a
directory, and the previous default was to configure one cache path.
There are no extant users of this role, though, so it doesn't really
matter.
The default set of access control lists and access rules for Squid are
fine for allowing hosts on the local network access to the web in
general. For other uses, such as web filtering, etc. more complex rules
may be needed. To that end, the *squid* role now supports some
additional variables. Notably, `squid_acl` contains a map of ACL names
to list entries and `squid_http_access` contains a list of access rules.
If these are set, their corresponding defaults are not included in the
rendered configuration file.
Unfortunately, the automatic transfer switch does not seem to work
correctly. When the standby source is a UPS running on battery, it does
*not* switch sources if the primary fails. In other words, when the
power is out and both UPS are running on battery, when the first one
dies, it will NOT switch to the second one. It has no trouble switching
when the second source is mains power, though, which is very strange.
I have tried messing with all the settings including nominal input
voltage, sensitivity, and frequency tolerence, but none seem to have any
effect.
Since it is more important for the machines to shut down safely than it
is to have an extra 10-15 minutes of runtime during an outage, the best
solution for now is to configure the hosts to shut down as soon as the
first UPS battery gets low. This is largely a waste of the second UPS,
but at least it will help prevent data loss.
*serial0.pyrocufflink.blue* has been replaced by
*serial1.pyrocufflink.blue*. The latter runs Fedora CoreOS and is
managed by the CUE-based configuration policy in *cfg.git*.
Recent versions of NUT have a *nut.target* unit that collects all of the
NUT-related services. Enabling any of the services individually does
effectively nothing, as it only adds the service as a `Wants` dependency
for *nut.target*, and that unit already has dependencies for all of
them. Thus, in order for the service to start at boot, *nut.target* has
to be enabled instead.
In situations where only *nut-monitor* should be enabled, enabling
*nut.target* is inappropriate, since that enables *nut-driver* and
*nut-server* as well. It's not clear why upstream made this change (it
was part of a [HUGE pull request][0]), but restoring the desired
behavior is easy enough by clearing the dependencies from *nut.target*.
Services that we want to start automatically can still be enabled
individually, and will start as long as *nut.target* is enabled.
[0]: https://github.com/networkupstools/nut/pull/330
The automatic transfer switch does not seem to work reliably when both
UPS sources are running on battery. This means all systems lose power
after the first UPS battery dies, even though the second UPS is still
online. To minimize the risk of data loss, at least until I figure out
what's wrong, I want both VM hosts to shut down as soon as the first UPS
signals that its battery is low.
Increasing the delay after starting the Kubernetes cluster to hopefully
allow things to "settle down" enough that starting services on follow up
VMs doesn't time out.
The Synapse server can sometimes take a very long time to start.
Increasing the start timeout should keep it from failing to come up when
the machine is under load.
Start Kubernetes earlier. Start Synapse later (it takes a long time to
start up and often times out when the VM hosts are under heavy load).
Start SMTP relay later as it's not really needed.
The UniFi controller service can sometimes take a really long time to
start up. This most frequently happens after a full outage, when the VM
hosts are very busy bringing everything up.
Dustin