The serial terminal server ("serterm") is a collection of scripts that
automate launching multiple `picocom` processes, one per USB-serial
adapter connected to the system. Each `picocom` process has its own
window in a `tmux` session, which is accessible via SSH on a dedicated
port (20022). Clients connecting to that SSH server will be
automatically attached to the `tmux` session, allowing them to access
the serial terminal server quickly and easily. The SSH server only
allows public-key authentication, so the authorized keys have to be
pre-configured.
In addition to automatically launching `picocom` windows for each serial
port when the terminal server starts, ports that are added (hot-plugged)
while the server is running will have windows created for them
automatically, by way of a udev rule.
Each `picocom` process is configured to log communications with its
respective serial port. This may be useful, for example, to find
diagnostic messages that may not be captured by the `tmux` scrollback
buffer.
All the stand-alone FCOS hosts now have Promtail running, forwarding
_systemd_ journal messages to Grafana Loki. The Kubernetes nodes will
have Promtail deployed as a Kubernetes pod.
I would really like to come up with a way to define variables for groups
of hosts, so that I do not have to add `promtail: prod.#promtail` to
every host's values file individually...
In order to simplify the process of adding new template render
instructions to all hosts, I've created a list of templates in the
`env/prod` module. This way, I only have to add templates there, and
all hosts that "inherit" from it will automatically get them.
I do not like how Fedora CoreOS configures `sudo` to allow the *core*
user to run privileged processes without authentication. Rather than
assign the user a password, which would then have to be stored
somewhere, we'll install *pam_ssh_agent_auth* and configure `sudo` to
use it for authentication. This way, only users with the private key
corresponding to one of the configured public keys can run `sudo`.
Naturally, *pam_ssh_agent_auth* has to be installed on the host system.
We achieve this by executing `rpm-ostree` via `nsenter` to escape the
container. Once it is installed, we configure the PAM stack for
`sudo` to use it and populate the authorized keys database. We also
need to configure `sudo` to keep the `SSH_AUTH_SOCK` environment
variable, so *pam_ssh_agent_auth* knows where to look for the private
keys. Finally, we disable the default NOPASSWD rule for `sudo`, if
and only if the new configuration was installed.
Dustin