dustin/basementhud/pipeline/head This commit looks goodDetails
By default, `systemd-socket-proxyd` keeps running after the remote
disconnects. This prevents the HUD control from reconnecting if e.g.
Firefox crashes.
`mqttdpms` is linked against OpenSSL 1.1. Rather than build multiple
copies that link to different versions, it's easier (for now) to just
install the old version of OpenSSL here.
Fedora's default PAM configuration assumes SSSD is installed. Since it
isn't, the system log is littered with errors about it:
> PAM unable to dlopen(/usr/lib64/security/pam_sss.so): /usr/lib64/security/pam_sss.so: cannot open shared object file: No such file or directory
> PAM adding faulty module: /usr/lib64/security/pam_sss.so
Because `tty1` is active when `xinit@user.service` starts on `tty7`,
Xorg cannot bind the modesetting driver to the DRM device:
> (II) systemd-logind: got fd for /dev/dri/card2 226:2 fd 13 paused 1
> (EE) Error systemd-logind returned paused fd for drm node
To fix this, we need to start Xorg on the active console.
Another potential fix would be to run `chvt 7` before running `startx`,
but running the GUI on tty1 seems to be what the major display managers,
e.g. GDM, are doing now.
dustin/basementhud/pipeline/head This commit looks goodDetails
I've switched Jenkins to use a Gitea Organization Folder so it can
auto-discover repositories owned by my Gitea user. This naturally
required the paths of existing projects to change. Here, we're updating
the path to the *mqttdpms* project to reflect its new location.
`depmod` needs to be run by `mkrootfs.sh` so that the kernel module
dependency information is included in the SquashFS image. Without this
information, `modprobe` cannot install kernel modules at runtime.
dustin/basementhud/pipeline/head This commit looks goodDetails
The `mqttdpms` tool will allow the screens to be turned off and on using
Home Assistant. We'll install it into the rootfs image by copying the
artifact from its Jenkins build into the overlay tree before building
the image.
basementhud/pipeline/head This commit looks goodDetails
The *hudctrl* service remotely controls Firefox on the Basement HUD
machine. It uses Firefox Marionette over TCP, and is itself controlled
by an HTTP API.
When the HUD machine starts up, it sends its monitor configuration to
*hudctrl* and tells it the display is ready to be controlled remotely.
For now, *hudctrl* is hosed on Rosalina. I will eventually move it to
the new metrics server, once it's built.
basementhud/pipeline/head This commit looks goodDetails
It turns out the Firefox marionette protocol can move/resize the Firefox
window, so `wmctrl` is not needed. We will need `xrandr`, though, to
help identify screen/monitor geometry, which we will send to the
control service so it knows how to position the Firefox windows.
The wonderful *ipapi.co* service can help identify the time zone of a
device based on the geolocation information for its IP address. We can
use this to set the `TZ` environment variable in the user session, which
Firefox will respect. This has the effect of showing the correct time
in e.g. Grafana dashboards, without hard-coding the time zone in the
rootfs image.
basementhud/pipeline/head This commit looks goodDetails
There's no reason to wait for an incoming Marionette connection to start
Firefox. Anyway, starting it earlier avoids a bunch of "connection
refused" spam from `systemd-socket-proxyd`.
Now that the system clock is set from the RTC, the time will be close
enough to correct by the time Xorg starts that there won't be any
certificate issues. We can shave several seconds off the startup time
now.
Dumb. It just enables the *systemd-timesyncd.service* in the initramfs,
which starts before the dracut initqueue, so there's no network yet.
This reverts commit 3efdfa9e54.
basementhud/pipeline/head This commit looks goodDetails
For some reason, `dhclient` fires an *EXPIRE* event every time it renews
the DHCP lease, even if the lease has not expired. `dhclient-script`
then removes the IP address from the interface, which of course breaks
the connection to the NBD server and blocks all reads from the root
filesystem. To avoid this strange situation, we need to stop
`dhclient-script` acting on the *EXPIRE* event. Luckily, it provides a
"hook" mechanism that we can use; since the hook script is dot-sourced,
we can just `exit` the process to avoid any further changes in that run.
basementhud/pipeline/head This commit looks goodDetails
This dumb module includes a ton of useless drivers. There doesn't seem
to be a way to avoid including it, because it is expressed as a
dependency of the *network* module, and even explicitly omitting the
modules it would include seems to have no effect.
*dracut* includes a ton of kernel modules in the initramfs by default.
These are totally unnecessary and make the image file huge, which makes
it take forever to load via TFTP. We can save a lot of space and time
by only including the specific drivers we need to mount the root
filesystem.
basementhud/pipeline/head This commit looks goodDetails
By default, the emergency shell prompts for the root password. If the
root account is locked, the emergency shell is unusable. This makes it
impossible to troubleshoot issues before the network is available.
Fortunately, systemd can be configured to skip the root password prompt
and allow access to the emergency shell with an environment variable.
basementhud/pipeline/head This commit looks goodDetails
NetworkManager is a memory hog. We don't need it to do anything except
renew the DHCP lease when it's time, so let's switch to a more
lightweight alternative. Unfortunately, Fedora does not include any
systemd units for `dhclient`, so we have to provide our own.
Running Voonex (Python), Playwight (Node.js), and Firefox simultaneously
takes way too much memory. Using the NBD swap prevents hard lockups and
crashes in OOM situations, but it makes the machine *incredibly* slow.
To avoid needing to push anything into the network swap, I want to try
and reduce the memory footprint as much as possible. To that end, we
will go back to running just Firefox. To control it, we will use the
Marionette protocol.
Firefox only allows control via Marionette over the loopback interface.
This is hard-coded in the Marionette server, and cannot be changed at
runtime. To allow a remote machine (running Voonex) to control it, we
need to expose the socket to the network. Fortunately, *systemd*
includes a tool for exactly this purpose: `systemd-socket-proxyd`.
Machine-specific data, such as SSH keys, should not be included in the
immutable root filesystem image, as this would prevent multiple machines
running from the same image. These few files can be stored locally, on
the SD card on eMMC flash.
The first time a machine boots up using this image, its local storage is
initialized. This involves creating a new filesystem on the block
device and generating SSH host keys. Once the storage is initialized,
it is remounted read-only. All subsequent mounts are read-only as well.
Using zram to compress pages did not provide enough memory to run
Firefox, Node, and Python for an extended period of time. Here's hoping
swap-on-NBD will be fast and reliable enough to work.
Note that systemd's default behavior with respect to swap areas is to
enable them all rather early in the boot process, before the network is
fully configured. As such, we have to use the *noauto* flag to disable
the default dependencies and add our own in a later target.
[Playwright] is a cross-browser automation framework, intended for
automating browser-based GUI application tests. It also works as a
general remote-control tool for automating basically anything having to
do with a browser.
Playwright requires a special Firefox binary that it manages itself. To
avoid downloading and installing Playwright and Firefox, and thus
storing them in memory, we need them to be already available in the
rootfs image. Since I want the control software to be updated easily,
without rebooting the machine, I decided to separate it into its own
project, [Voonex]. A short shell script to install/update and run it is
launched by the systemd user instance.
[Playwright]: https://playwright.dev/
[Voonex]: https://git.pyrocufflink.blue/dustin/voonex
While developing, it will be nice to have the serial console available
for troubleshooting, especially for issues that prevent network/SSH from
working correctly. I've temporarily connected a USB-TTL console cable
to *serial0* to control the serial console of the HUD machine.
Using `--no-W` (disable whole file transfer) saves some time and
bandwidth when uploading the root filesystem image, but it has the
negative side-effect of overwriting the destination file in place. If
any NBD clients are currently running using the image as their root
filesystem, they are likely to experience SquashFS corruption errors, as
the filesystem driver does not expect the underlying data to change once
it has been mounted. As such, we have to use the default file transfer
method to create a new file and atomically replace the old file once the
transfer is complete. The original file will be unlinked and will
eventually be deleted once no clients are using it.
Matchbox Window Manager, being designed for mobile devices, does not
handle multiple monitors well. It manages the monitors together as one
giant screen, which will make it extremely difficult to manage separate
Firefox windows on each screen. Openbox is almost as lightweight as
Matchbox, but it works significantly better for this use case.
Dustin