The *base* role will now set the password for the *root* user, if the
`root_password_hash` variable is defined. This ensures that there is a
way to log into machines directly, even if other authentication
mechanisms like Active Directory are unavailable.
Occasionally, VMs running on the main libvirt VM hosts will freeze or
otherwise become unavailable via network. Sometimes, when this happens,
their normal consoles are unresponsive as well. Having the serial
console available as a fallback can sometimes be helpful in recovering
from such situations.
To ensure the serial console is available on all VMs, we use a "dynamic"
group, based on the virtualization type and role of the managed node.
All KVM-based virtual machines are included in a group named *kvm-vm*.
A play in `base.yml` applies the *serial-console* role to members of
this group.
The *serial-console* Ansible role enables and starts a systemd service
unit to activate a console getty on the specified serial console device
(by default: ttyS0). This is particularly useful for virtual machines,
allowing one to control them in absence of a graphical VM management
tool.
The `hassdb.yml` playbook is no longer used; the new Home Assistant
deployment uses the built-in database again, since it is stored on NVMe
instead of an SD card.
Further, the current deployment is hosted by a machine with a single
filesystem, which thus cannot be remounted read-only after applying
policy.
Some playbooks apply only to hosts that do not have read-only root
filesystems. For these, the `rw_limit` pattern will be empty. The
*Remount R/W* and *Remount R/O* stages should be skipped when this is
the case.
Filesystems like NFS and CIFS require "helper" utilities (i.e.
`mount.nfs` and `mount.cifs`, respectively). These need to be installed
in order for a system to be able to mount those filesystems.
The current shared storage system uses NFSv4, and as such, the
*nfs-utils* package needs to be installed on the VM hosts.
With the transition away from *dhcpcd* on the VM hosts, there is no
longer any need for a custom wait script that must run prior to
attempting to mount the shared filesystem. This dramatically simplifies
the configuration necessary for shared storage.
I don't really see any reason why the shared storage configuration needs
to be managed by a separate role. The *vmhost* role is not really
generic anyway, and will probably not work for any other VM host
deployment besides the two machines running now. As such, I think it
makes sense to move the task to mount the shared filesystem into the
*vmhost* role and drop the *dch-storage-net* role.
The *libvirt-daemon-driver-network* package provides support for
managing virtual networks with libvirt. It is necessary in order to use
managed networks in VM configuration, as opposed to directly specifying
VM network interfaces in their domain configuration.
*vmhost0.pyrocufflink.blue* no longer uses `dhcpcd` for network
configuration, but *systemd-networkd*.
The host-specific network settings for a VM host include the
configuration for the management interface, as well as the configuration
of the physical ports that make up the bonded interfaces.
Originally, the network configuration for the VM networks and the
storage network was configured using the *netifaces* role. This has
effectively stopped working in recent versions of Fedora, as it sort of
relied on `dhcpcd`, which has not been maintained in Fedora for a while
and no longer behaves correctly. After evaluating *NetworkManager* as a
replacement, I decided that *systemd-networkd* is a more appropriate
solution.
There are effectively two "layers" of network configuration needed for
the VM hosts: the host-specific settings, and the common settings. The
host-specific settings include such properties as the IP address of the
management interface and the names of the physical ports that make up
the bonded interfaces. The common settings are the bonded interfaces,
the VLAN interfaces created on top of the bond, and the bridges that
provide access to VMs.
To configure the host-specific settings, each host simply needs the
appropriate `networkd_*` variables in its `host_vars` file. For the
common settings, we apply the *systemd-networkd* role again in the
`vmhost.yml` with different values for these variables. Thus,
effectively, `systemd-networkd.yml` manages the host-specific settings,
while `vmhost.yml` manages the common settings.
*systemd-networkd* is (currently) my preferred way to manage network
interfaces on machines running Fedora. The *systemd-networkd* role
provides a generic way to configure network links, devices, and
interfaces, using Ansible variables to generate network unit
configuration files.
*hass1.pyrocufflink.blue* and *hassdb0.pyrocufflink.blue* were part of
the old Home Assistant deployment. Everything has been migrated to
*hass2.pyrocufflink.blue*, so these machines can be decommissioned now.
I couldn't get RTMP to work on the Back Yard camera because the `ffmpeg`
process kept crashing:
```
ffmpeg.back_yard.clips_rtmp ERROR : av_interleaved_write_frame(): Connection reset by peer
ffmpeg.back_yard.clips_rtmp ERROR : [flv @ 0x5562090c8ec0] Failed to update header with correct duration.
ffmpeg.back_yard.clips_rtmp ERROR : [flv @ 0x5562090c8ec0] Failed to update header with correct filesize.
ffmpeg.back_yard.clips_rtmp ERROR : Error writing trailer of rtmp://127.0.0.1/live/back_yard: Connection reset by peer
watchdog.back_yard INFO : Terminating the existing ffmpeg process...
watchdog.back_yard INFO : Waiting for ffmpeg to exit gracefully...
```
I thought increasing the value of `--shm-size` argument for `podman`
would help, but even going as high as 1024 mebibytes did not resolve the
problem.
Ultimately, I decided that it is not really necessary to view the full
4k stream in real time. The back yard camera supports three streams, so
I set them all up for different roles. I briefly considered using a
single 1080p stream for both object detection and RTMP streaming, but
this consumed considerable CPU time, so I decided against it for now. I
may re-evaluate that option if I decide to purchase a TPU.
The `collectd_df` variable can be used to configure the *df* plugin for
collectd. It should contain a map on key-value pairs that correspond
exactly to the plugin's configuration options.
*nvr0.pyrocufflink.blue* hosts Frigate. It is deployed on a separate
subnet, for two reasons:
* To avoid streaming video from the cameras through the firewall
* To prevent any hosts on the LAN except Home Assistant from
communicating with Frigate, since it does not have any kind of
authentication or access control
Frigate is an NVR that uses machine learning to detect objects on camera
in real time. It integrates with Home Assistant to expose sensors which
can be used for automation, etc.
The only official way to deploy Frigate is with a container, so we use
Podman and systemd to manage it.
For hosts that cannot send metrics via multicast (e.g. because they are
on a different subnet), *collectd* needs to listen on the all-hosts
unicast address.
The VM hosts have multiple network interfaces with IPv6 addresses, so
collectd may not always choose the correct one to send metrics. Thus we
have to explicitly tell it to use the management interface, to avoid it
sending data on the SAN interface.
The production deployment of *dnsmasq* for Home Assistant has deviated
from how the *hass-dhcp* role configures it. Bringing the role back in
sync with how things really are.
ZwaveJS2Mqtt includes a very powerful web-based UI for configuring and
controlling the Z-Wave network. This functionality is no longer
available within Home Assistant itself, so being able to access the
ZwaveJS2Mqtt UI is crucial to operating the network.
I wanted to make the UI available at */zwave/*, which requires using
*mod_rewrite* to conditionally proxy requests based on the `Connection`
HTTP header, since the UI passes both HTTP and WebSocket requests to the
same paths. *mod_rewrite* configuration is not inherited from the main
server configuration to virtual hosts, so the
`RewriteRule`/`RewriteCond` directives have to be specified within the
`<VirtualHost>` block. This means that the Home Assistant proxy
configuration has to be within its own virtual host, and the
Zwavejs2Mqtt configuration has to be there as well.
*hass2.pyrocufflink.blue* is a Raspberry Pi Compute Module 4-based
system, currently mounted in a WaveShare CM4 Mini Base Board (A). With
an NVMe SSD for primary storage, it runs significantly faster than a
standard Raspberry Pi 4, and blows the old Raspberry Pi 3-based Home
Assistant deployment out of the water. It has a Zooz 700 series Z-Wave
Plus S2 USB stick and a ConBee II Zigbee USB stick attached to its USB
2.0 ports. It runs a customized Fedora Minimal distribution.
Zigbee2MQTT is very similar to ZwaveJS2Mqtt: it is a daemon process that
communicates with the Zigbee radio and integrates with Home Assistant
using MQTT. Naturally, I decided to deploy it in the same way as
ZwaveJS2Mqtt, using a systemd unit to run it in a container with Podman.
Mosquitto 2.x included two significant changes from 1.6:
* There is no longer a "default" listener; all listeners are configured
in the same way
* The daemon drops privileges *before* reading TLS certificates and
private keys
Home Assistant no longer recommends using the built-in libopenzwave
integration for communicating with Z-Wave devices. Evidently, OpenZWave
is no longer maintained, and community efforts have shifted toward
Z-Wave JS.
Z-Wave JS is architecturally much different than the legacy Z-Wave
integration. Instead of running the network controller inside the Home
Assistant process, a separate daemon communicates with the Z-Wave radio.
Home Assistant integrates with that daemon using a WebSockets API. This
has the advantage of decoupling the network operation from the lifecycle
of the Home Assistant process: restarting Home Assistant (e.g. to load
new configuration changes) does not take the Z-Wave network offline.
ZwaveJS2Mqtt is a distribution of the Z-Wave JS daemon, as well as a
web-based user interface for configuring it. Although its name implies
that it uses MQTT for communication, this feature is actually optional,
and the native WebSockets API can still be used for integration with
Home Assistant.
I decided to follow the same deployment pattern for ZwaveJS2Mqtt as for
Home Assistant itself: run the application from a container image using
Podman. This of course simplifies the installation of the application
significantly, leaving most of that work up to the maintainer of the
container image. Podman provides the container runtime, managing the
privileges, etc. The systemd service unit starts Podman, configuring an
ephemeral container on each run. The container uses the default network
namespace, avoiding the unnecessary overhead of port mapping. It uses
Podman's "rootless" mode, via the `--uidmap` and `--gidmap` arguments,
mapping users inside the container, including root, to unprivileged
users on the host. The Z-Wave radio, which is specified by the
`zwavejs_device` Ansible variable, is passed into the container via the
`--device` argument.
Installing Home Assistant in a Python virtualenv is rather tedious,
especially on non-x86 machines. The main issue is Python packages that
include native extensions, as many of these do not have binary wheels
available for aarch64, etc. on PyPI. Thus, to install these, they have
to be built from source, which then requires the appropriate development
packages to be installed. Additionally, compiling native code on a
Raspberry Pi is excruciatingly slow. I have considered various ways of
mitigating this, but all would require a substantial time investment,
both up front and ongoing, making them rather pointless. Eventually, I
settled on just deploying the official Home Assistant container image
with Podman.
Although Podman includes a tool for generating systemd service unit
files for running containers, I ended up creating my own for several
reasons. First and foremost, the generated unit files configure the
containers to run as *root*, but I wanted to run Home Assistant as an
unprivileged user. Unfortunately, I could not seem to get the container
to work when dropping privileges using the `User` directive of the unit.
Fortunately, `podman` has `--uidmap` and `--gidmap` arguments, which I
was able to use to map UID/GID 0 in the container to the *homeassistant*
user on the host. Another drawback of the generated unit files is that
they specify a "forking" type service, which is not really necessary.
Podman/conmon supports the systemd notify protocol, but the generator
has not been updated to make use of that yet.
Recent versions of Home Assistant are more strict with respect to how
reverse proxies are handled. In order to use one, it must be explicitly
listed in the configuration file. Therefore, the *homeassistant*
Ansible role will now create a stub `configuration.yaml`, based on the
one generated by Home Assistant itslf when it starts for the first time
on a new machine, that includes the appropriate configuration for a
reverse proxy running on the same machine. The stub configuration will
not overwrite an existing configuration file, so it is only useful when
deploying Home Assistant for the first time on a new machine.
Overall, although I think a 300+ MB container image is ridiculous,
deploying Home Assistant this way should make it a lot easier to manage,
especially when updating.
Zezere is the Fedora IoT device provisioning service. It is the
software that runs *provision.fedoraproject.org*, but it can be
self-hosted (if you can figure it out; there is no documentation
whatsoever).
The main use case for running Zezere locally is to automatically add
trusted SSH public keys to Fedora IoT devices, without depending on a
cloud service. This playbook sets up Zezere with the very minimal
configuration needed to meet this goal.
Although configuration policy is not yet available for Prometheus
itself, the `collectd.yml` playbook also uses the *prometheus* host
group. Specifically, hosts in this group are configured to receive
collectd data from other hosts and expose those data through the
`write_prometheus` plugin.
This commit introduces the *grafana* role and the corresponding
`grafana.yml` playbook. The role installs Grafana using the system
package manager, and configures the server (including LDAP
authentication).
Occasionally, ProtonVPN servers randomly reject the EAP authentication
credentials. When this happens, the tunnel fails and is not restarted
automatically by strongSwan. As such, the watchdog needs to react to
this event as well.
Since the Nextcloud configuration file is managed by the configuration
policy, all of the settings configurable through the web UI need to be
templated. One important group of settings is the outbound email
configuration. This can now be configured using the `nextcloud_smtp`
Ansible variable.
Dustin