We're going to run MinIO on the BURP server to provide a backup target
for the [Postgres Operator][0]/[WAL-E][1]. Although the Postgres
Operator also supports backups via [WAL-G][2], which supports more
backup targets like SFTP, the operator does not support restoring from
those targets. As such, the best way to get fully-featured backups for
the Postgres Operator, including environment cloning, etc., is to use
S3. Since I absolutely do not want to store my backups "in the cloud,"
using MinIO seems a decent alternative. Running it on the BURP server
allows the backups to be stored and rotated along with regular system
backups.
[0]: https://github.com/zalando/postgres-operator/
[1]: https://github.com/wal-e/wal-e
[2]: https://github.com/wal-g/wal-g
The BURP storage volume is now backed by a Linux MD RAID array, so we
want to monitor its state. Furthermore, since this machine is a
physical device, we should monitor its thermal characteristics as well.
I moved the metrics Pi from the red network to the blue network. I
started to get uncormfortable with the firewall changes that were
required to host a service on the red network. I think it makes the
most sense to define the red network as egress only.
*mtrcs0.pyrocufflink.red* is a Raspberry Pi CM4 on a Waveshare
CM4-IO-BASE-B carrier board with a NVMe SSD. It runs a custom OS built
using Buildroot, and is not a member of the *pyrocufflink.blue* AD
domain.
*mtrcs0.p.r* hosts Victoria Metrics/`vmagent`, `vmalert`, AlertManager,
and Grafana. I've created a unique group and playbook for it,
*metricspi*, to manage all these applications together.
It seems with each new release of Fedora, some feature or other of
*collectd* gets broken. In Feodra 36, the *interfaces* plugin does not
seem to work reliably, and the *md* plugin logs a *lot* of errors.
While these issues are investigated upstream, we either need to manage
our own policy for collectd or mark the `collectd_t` domain permissive.
I chose the latter because I'm lazy and I don't consider collectd to be
that big of a threat to security.
*nvr1.pyrocufflink.blue* is the new video recording server. It is a
1U rack-mounted physical machine based on the [Jetway
JBC150F596-3160-B][0] barebone system. It replaces
*nvr0.pyrocufflink.blue* in this role.
[0]: https://www.jetwaycomputer.com/JBC150F596.html
Using *systemd-networkd* to configure network interfaces on *vmhost0* is
working really well. It is decidedly more stable than *dhcpcd* was, and
certainly easier to work with than NetworkManager. Let's go ahead and
switch *vmhost1* as well.
Transitioning from push-based to pull-based monitoring with
Prometheus/collectd. The *write_prometheus* plugin will be installed on
all hosts, and Prometheus will be configured to scrape them directly.
*vmhost0.pyrocufflink.blue* no longer uses `dhcpcd` for network
configuration, but *systemd-networkd*.
The host-specific network settings for a VM host include the
configuration for the management interface, as well as the configuration
of the physical ports that make up the bonded interfaces.
*nvr0.pyrocufflink.blue* hosts Frigate. It is deployed on a separate
subnet, for two reasons:
* To avoid streaming video from the cameras through the firewall
* To prevent any hosts on the LAN except Home Assistant from
communicating with Frigate, since it does not have any kind of
authentication or access control
The VM hosts have multiple network interfaces with IPv6 addresses, so
collectd may not always choose the correct one to send metrics. Thus we
have to explicitly tell it to use the management interface, to avoid it
sending data on the SAN interface.
*hass2.pyrocufflink.blue* is a Raspberry Pi Compute Module 4-based
system, currently mounted in a WaveShare CM4 Mini Base Board (A). With
an NVMe SSD for primary storage, it runs significantly faster than a
standard Raspberry Pi 4, and blows the old Raspberry Pi 3-based Home
Assistant deployment out of the water. It has a Zooz 700 series Z-Wave
Plus S2 USB stick and a ConBee II Zigbee USB stick attached to its USB
2.0 ports. It runs a customized Fedora Minimal distribution.
Because *hass1.pyrocufflink.blue* has multiple interfaces, collectd does
not know which interface it should use to send multicast metrics
messages. To force it to use the wired interface, which is connected to
the default internal ("blue") network, the `interface` property needs to
be set.
BIND response policy zones (RPZ) support provides a mechanism for
overriding the responses to DNS queries based on a wide range of
criteria. In the simplest form, a response policy zone can be used to
provide different responses to different clients, or "block" some DNS
names.
For the Pyrocufflink and related networks, I plan to use an RPZ to
implement ad/tracker blocking. The goal will be to generate an RPZ
definition from a collection of host lists (e.g. those used by uBlock
Origin) periodically.
This commit introduces basic support for RPZ configuration in the
*named* role. It can be activated by providing a list of "response
policy" definitions (e.g. `zone "name"`) in the `named_response_policy`
variable, and defining the corresponding zones in `named_zones`.
DHCP is provided by *dns1.pyrocufflink.blue* now, not the gateway. To
allow dynamic DNS updates from it, the correct source address must be
listed in the zone configuration for *pyrocufflink.red*.
Because *jenkins0.pyrocufflink.blue* runs Docker, it has an extra
virtual interface and IP address, for container communication. By
default, Samba registers all IP addresses in DNS, and cannot
differentiate between the actual interface and the Docker bridge. This
can cause other hosts to attempt to contact *jenkins0.pyrocufflink.blue*
using the wrong address.
The `samba_interfaces` variable controls the value of the `interfaces`
global configuration option for Samba. One of the things this option
controls is which addresses to register in DNS. By setting it to the
network address of the *pyrocufflink.blue* network, we can prevent the
virtual address from being used at all.
To avoid having a single point of failure, a second recursive DNS server
is necessary. This will be useful in cases where the VM hosts must both
be taken offline, but Internet access is still required.
The new server, *dns1.pyrocufflink.blue*, has all the same zones defined
as the original. It forwards the *pyrocufflink.blue* zone and
corresponding reverse zones to the domain controllers, and acts as a
slave for the *pyrocufflink.red* zone.
In order to support adding a second DNS server, the BIND zone
configuration needs to be partially modularized. While the forwarder
definitions for *pyrocufflink.blue*, etc. will remain the same, the
*pyrocufflink.red* zone will be different, as it will be a slave zone on
the second server. This commit breaks up the definition of the
`named_zones` variable into two parts:
* `pyrocufflink_red_zones`: This is a list of zone objects for
*pyrocufflink.red* and its corresponding reverse zone. On
*dns1.pyrocufflink.blue*, these are master zones. On the new server,
these will be slaves.
* `pyrocufflink_common_zones`: This is a list of zone objects for the
zones that are the same on both servers, since they are all forwarding
zones.
Similarly, the `named_keys` variable only needs to be defined on the
master, since DHCP will only send updates there.
*smtp1.pyrocufflink.blue* is a VM that will replace
*smtp0.pyrocufflink.blue*, a Raspberry Pi.
I decided that there is little use in having the availability guarantee of
a discreet machine for the SMTP relay. The only system that would NEED
to send mail if the VM host fails is Zabbix, which operates as its own
relay anyway. As such, the main relay can be a VM, and the Raspberry Pi
can be repurposed as a recursive DNS server.
The Management network (VLAN 10, 172.30.0.240/28) will be used for
communication with and configuration of network devices including
switches and access points. This keeps configuration separate from
normal traffic, and allows complete isolation of infrastructure devices.
The gateway device is now monitored by Zabbix. Adding it to the *zabbix*
group ensures that the Zabbix agent is installed and configured
correctly.
Because the *zabbix-agent* role has a task to configure FirewallD, the
`host_uses_firewalld` variable needs to be set to `false` for *gw0*,
since it does not use FirewallD.
The `gitea_root_url` variable is used to configure the root URL for
Gitea, which is in turn used to generate HTTP/HTTPS "clone" links for
Git repositories. If this value is not set, the default is used, which
does not work since the application is behind a reverse proxy.
Dustin