_systemd_ automatically recursively changes the ownership of the paths
listed in `StateDirectory` when the unit is activated. This can take a
very long time, as the Loki storage directory contains hundreds of
thousands of files. Since we also have `podman` change the ownership,
that *doubles* the time taken. Similarly, with `podman` also configured
to change the SELinux label of the files in that path, even more time is
wasted at startup.
To avoid all these time wasters, we need to avoid having _systemd_
manage the state directory and create it with the proper ownership and
SELinux label manually. Here, we're only manipulating the metadata of
the top-level directory; anything within the directory is untouched.
This ensures that the directory is always there and has the correct
permissions, but does not spend any time changing anything that doesn't
need changed.
[Frigate] is an open source network video recording software with
advanced motion detection using machine learning object detection. It
uses `ffmpeg` to stream video from one or more RTSP-capable IP video
cameras and passes the images through an object detection process. To
improve the performance of the machine learning model, it supports using
a Coral EdgeTPU device, which requires special drivers: `gasket` and
`apex`.
Frigate is configured via a (rather compex) YAML document, some of the
schema of which is modeled in `schema.cue` (the parts I need, anyway).
[Frigate]: https://frigate.video/
The serial terminal server ("serterm") is a collection of scripts that
automate launching multiple `picocom` processes, one per USB-serial
adapter connected to the system. Each `picocom` process has its own
window in a `tmux` session, which is accessible via SSH on a dedicated
port (20022). Clients connecting to that SSH server will be
automatically attached to the `tmux` session, allowing them to access
the serial terminal server quickly and easily. The SSH server only
allows public-key authentication, so the authorized keys have to be
pre-configured.
In addition to automatically launching `picocom` windows for each serial
port when the terminal server starts, ports that are added (hot-plugged)
while the server is running will have windows created for them
automatically, by way of a udev rule.
Each `picocom` process is configured to log communications with its
respective serial port. This may be useful, for example, to find
diagnostic messages that may not be captured by the `tmux` scrollback
buffer.
I discovered today that if anonymous Grafana users have Viewer
permission, they can use the Datasource API to make arbitrary queries
to any backend, even if they cannot access the Explore page directly.
This is documented ([issue #48313][0]) as expected behavior.
I don't really mind giving anonymous access to the Victoria Metrics
datasource, but I definitely don't want anonymous users to be able to
make Loki queries and view log data. Since Grafana Datasource
Permissions is limited to Grafana Enterprise and not available in
the open source version of Grafana, the official recommendation from
upstream is to use a separate Organization for the Loki datasource.
Unfortunately, this would preclude having dashboards that have graphs
from both data sources. Although I don't have any of those right now, I
like the idea and may build some eventually.
Fortunately, I discovered the `send_user_header` Grafana configuration
option. With this enabled, Grafana will send an `X-Grafana-User` header
with the username of the user on whose behalf it is making a request to
the backend. If the user is not logged in, it does not send the header.
Thus, we can detect the presence of this header on the backend and
refuse to serve query requests if it is missing.
[0]: https://github.com/grafana/grafana/issues/48313
Grafana Loki explicitly eschews built-in authentication. In fact, its
[documentation][0] states:
> Operators are expected to run an authenticating reverse proxy in front
> of your services.
While I don't really want to require authentication for agents sending
logs, I definitely want to restrict querying and viewing logs to trusted
users.
There are _many_ reverse proxy servers available, and normally I would
choose _nginx_. In this case, though, I decided to try Caddy, mostly
because of its built-in ACME support. I wasn't really happy with how
the `fetchcert` system turned out, particularly using the Kubernetes API
token for authentication. Since the token will eventually expire, it
will require manual intervention to renew, thus mostly defeating the
purpose of having an auto-renewing certificate. So instead of using
_cert-manager_ to issue the certificate and store it in Kubernetes, and
then having `fetchcert` download it via the Kubernetes API, I set up
_step-ca_ to handle issuing the certificate directly to the server. When
Caddy starts up, it contacts _step-ca_ via ACME and handles the
challenge verification automatically. Further, it will automatically
renew the certificate as necessary, again using ACME.
I didn't spend a lot of time optimizing the Caddy configuration, so
there's some duplication there (i.e. the multiple `reverse_proxy`
statements), but the configuration works as desired. Clients may
provide a certificate, which will be verified against the trusted issuer
CA. If the certificate is valid, the client may access any Loki
resource. Clients that do not provide a certificate can only access the
ingestion path, as well as the "ready" and "metrics" resources.
[0]: https://grafana.com/docs/loki/latest/operations/authentication/
The Promtail container image is pretty big, so it takes quite some time
to pull on a slow machine like a Raspberry Pi. Let's increase the
startup timeout so the service is less likely to fail while the image is
still being pulled.
[Promtail][0] is the log collection agent for Grafana Loki. It reads
logs from various locations, including local files and the _systemd_
journal and sends them to Loki via HTTP.
Loki configuration is a highly-structured YAML document. Thus, instead
of using Tera template syntax for loops, conditionals, etc., we can use
the full power of CUE to construct the configuration. Using the
`Marshal` function from the built-in `encoding/yaml` package, we
serialize the final configuration structure as a string and write it
verbatim to the configuration file.
I have modeled most of the Promtail configuration schema in the
`du5t1n.me/cfg/app/promtail/schema` package. Having the schema modeled
will ensure the generated configuration is valid during development
(i.e. `cue export` will fail if it is not), which will save time pushing
changes to machines and having Loki complain.
The `#promtail` "function" in `du5t1n.me/cfg/env/prod` makes it easy to
build our desired configuration. It accepts an optional `#scrape`
field, which can be used to provide specific log scraping definitions.
If it is unspecified, the default configuration is to scrape the systemd
journal. Hosts with additional needs can supply their own list,
probably including the `promtail.scrape.journal` object in it to get the
default journal scrape job.
[0]: https://grafana.com/docs/loki/latest/send-data/promtail/
According to the [Grafana Loki documentation][0], sending SIGHUP to the
Loki process will instruct it to reload its configuration. This is
necessary in order for it to re-read its server certificate after it has
been renewed.
[0]: https://grafana.com/docs/loki/latest/configure/#reload-at-runtime
Before going into production with Grafana Loki, I want to set it up to
use TLS. To that end, I have configured _cert-manager_ to issue it a
certificate, signed by _DCH CA_. In order to use said certificate,
we need to configure `fetchcert` to run on the Loki server.
The `fetchcert` tool is a short shell script that fetches an X.509
certificate and corresponding private key from a Kubernetes Secret,
using the Kubernetes API. I originally wrote it for the Frigate server
so it could fetch the _pyrocufflink.blue_ wildcard certificate, which is
managed by _cert-manager_. Since then, I have adapted it to be more
generic, so it will be useful to fetch the _loki.pyrocufflink.blue_
certificate for Grafana Loki.
Although the script is rather simple, it does have several required
configuration parameters. It needs to know the URL of the Kubernetes
API server and have the certificate for the CA that signs the server
certificate, as well as an authorization token. It also needs to know
the namespace and name of the Secret from which it will fetch the
certificate and private key. Finally, needs to know the paths to the
files where the fetched data will be written.
Generally, after certificates are updated, some action needs to be
performed in order to make use of them. This typically involves
restarting or reloading a daemon. Since the `fetchcert` tool runs in
a container, it can't directly perform those actions, so it simply
indicates via a special exit code that the certificate has been updated
and some further action may be needed. The
`/etc/fetchcert/postupdate.sh` script is executed by _systemd_ after
`fetchcert` finishes. If the `EXIT_STATUS` environment variable (which
is set by _systemd_ to the return code of the main service process)
matches the expected code, the configured post-update actions will be
executed.
When _collectd_ calls *statvfs(3)* on paths like
`/host/proc/sys/fs/binfmt_misc` which are configured for auto-mounting,
_systemd_ logs hundreds of messages like these:
```
systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 1303 (reader#3)
systemd[1]: proc-sys-fs-binfmt_misc.automount: Automount point already active?
```
Eventually, _collectd_ logs an error:
```
collectd[1132]: statvfs(/host/proc/sys/fs/binfmt_misc) failed: Too many levels of symbolic links
```
This happens on every scrape interval.
To avoid this, we can configure _collectd_ to skip calling *statvfs(3)*
on _autofs_ mount points. Even if it did work correctly, we wouldn't
really want _collectd_ triggering automounts; that would pretty much
defeat the purpose of them.
Since *systemd* starts the *reload-udev-rules.service* unit as soon as
any file in the `/run/containers/udev-rules` directory changes, the `cp`
command may start before all of the files have been copied out of the
container. If this happens, some of the rules will not get copied to
the final path, and thus will not be processed by *udev*.
Togive the container a chance to finish copying all of the files before
we process them, we need a bit of a delay. Obviously, this is not a
perfect solution, as it could potentially take longer than 250ms to copy
the files in some cases, but hopefully those cases are rare enough to
not worry about.
I do not like how Fedora CoreOS configures `sudo` to allow the *core*
user to run privileged processes without authentication. Rather than
assign the user a password, which would then have to be stored
somewhere, we'll install *pam_ssh_agent_auth* and configure `sudo` to
use it for authentication. This way, only users with the private key
corresponding to one of the configured public keys can run `sudo`.
Naturally, *pam_ssh_agent_auth* has to be installed on the host system.
We achieve this by executing `rpm-ostree` via `nsenter` to escape the
container. Once it is installed, we configure the PAM stack for
`sudo` to use it and populate the authorized keys database. We also
need to configure `sudo` to keep the `SSH_AUTH_SOCK` environment
variable, so *pam_ssh_agent_auth* knows where to look for the private
keys. Finally, we disable the default NOPASSWD rule for `sudo`, if
and only if the new configuration was installed.
Setting `AutoUpdate=registry` will tell Podman to automatically fetch
an updated container image from its corresponding registry and restart
the container. The `podman-auto-update.timer` systemd unit needs to be
active for this to happen on a schedule.
`dest` is not a valid option for the `--mount` argument to `podman`. To
specify where the target path, only `target`, `destination`, and `dst`
are valid.
`upsmon` is the component of NUT that tracks the status of UPSs and
reacts to their changing by sending notifications and/or shutting down
the system. It is a networked application that can run on any system;
it can run on a different system than `upsd`, and indeed can run on
multiple systems simultaneously.
Each system that runs `upsmon` will need a username and password for
each UPS it will monitor. Using the CUE [function pattern][0], I've
made it pretty simple to declare the necessary values under
`nut.monitor`.
[0]: https://cuetorials.com/patterns/functions/
*collectd* logs to syslog, so its output is lost when it's running in a
container. We can capture messages from it by mounting the journald
syslog socket into the container.
The `/run/udev/rules.d` directory may not always exist, especially at
boot. We need to ensure that it does before we try to copy rules
exported by containers into it, or the unit will fail.
Even with *collectd* configured to report filesystem usage by device, it
still only reports filesystems that are mounted (in its namespace).
Thus, in order for it to report filesystems like `/boot`, these need to
be mounted in the container.
I keep going back-and-forth on whether or not collectd should run in a
container on Fedora CoreOS machines. On the one hand, running it
directly on the host allows it to monitor filesystem usage by mount
point, which is consistent with how non-FCOS machines are monitored.
On the other hand, installing packages on FCOS with `rpm-ostree` is a
nightmare. It's _incredibly_ slow. There's also occasionally issues
installing packages if the base layer has not been updated in a while
and the new packages require an existing package to be updated.
For the NUT server specifically, I have changed my mind again: the
*collectd-nut* package depends on *nut-client*, which in turn depends on
Python. I definitely want to avoid installing Python on the host, but I
do not want to lose the ability to monitor the UPSs via collectd. Using
a container, I can strip out the unnecessary bits of *nut-client* and
avoid installing Python at all. I think that's worth having to monitor
filesystem usage by device instead of by mount point.
The only privilege NUT needs is access to the USB device nodes. Using a
device CGroup rule to allow this is significantly better than disabling
all restrictions. Especially since I discovered that `--privileged`
implies `--security-opt label=disable`, effectively disabling SELinux
confinement of the container.
NUT needs some udev rules in order to set the proper permissions on USB
etc. devices so it can run as an otherwise unprivileged user. Since
udev rules can only be processed on the host, these rules need to be
copied out of the container and evaluated before the NUT server starts.
To enable this, the *nut-server* container image copies the rules it
contains to `/etc/udev/rules.d` if that directory is a mount point. By
bind mounting a directory on the host at that path, we can get a copy of
the rules files outside the container. Then, using a systemd path unit,
we can tell the udev daemon to reload and reevaluate its rules.
SELinux prevents processes in containers from writing to
`/etc/udev/rules.d` directly, so we have to use an intermediate location
and then copy the rules files to their final destination.
Dustin