The `config.yml` document for *kitchen* contains several "secret" values
(e.g. passwords to Nextcloud, MQTT, etc.). We don't want to commit
these to the Git repository, of course, but as long as Kustomize expects
to find the `config.yml` file, we won't be able to manage the
application with Argo CD. Ultimately, *kitchen* needs to be modified to
be able to read secrets separately from config, but until then, we will
have to avoid managing `config.yml` with Kustomize.
I actually created this a long time ago, but forgot to update the
manifest in Git.
The *homeassistant* database is used by Home Assistant for its
*recorder* component, which stores long-term statistics. The data
stored here are only used for e.g. History and Logbook; current entity
states are still stored on the filesystem.
The `argocd` command needs to have its own OIDC client configuration,
since it works like a "public" client. To log in, run
```sh
argocd login argocd.pyrocufflink.blue --sso
```
Without `disableNameSuffixHash` enabled, Kustomize will create a unique
ConfigMap any time the contents of source file change. It will also
update any Deployment, StatefulSet, etc resources to point to the new
ConfigMap. This has the effect of restarting any pods that refer to the
ConfigMap whenever its contents change.
I had avoided using this initially because Kustomize does *not* delete
previous ConfigMap resources whenever it creates a new one. Now that we
have Argo CD, though, this is not an issue, as it will clean up the old
resources whenever it synchronizes.
[Argo CD] is a Kubernetes-native GitOps/continuous deployment manager.
It monitors the state of Kubnernetes resources, such as Pods,
Deployments, ConfigMaps, Secrets, and Custom Resources, and synchronizes
them with their canonical definitions from a Git repository.
*Argo CD* consists of various components, including a Repository
Service, an Application Controller, a Notification Controller, and an
API server/Web UI. It also has some optional components, such as a
bundled Dex server for authentication/authorization, and an
ApplicationSet controller, which we will not be using.
[Argo CD]: https://argo-cd.readthedocs.io/
[Sealed Secrets] will allow us to store secret values in the Git
repository, since the actual secrets are encrypted and can only be
decrypted using the private key stored in the Kubernetes cluster.
I have been looking for a better way to deal with secrets for some time
now. For one thing, having the secret files ignored by Git means they
only exist on my main desktop. If I need to make changes to an
application from another machine, I have to not only clone the
repository, but also manually copy the secret files. That sort of
makes my desktop a single point-of-failure. I tried moving all the
secret files to another (private) repository and adding it as a
submodule, but Kustomize did not like that; it will only load files from
the current working directory, or another Kustomize project. Having to
create two projects for each application, one for the secrets and one
for everything else, would be tedious and annoying. I also considered
encrypting all the secret files with e.g. GnuPG and creating Make
recipies for each project to decrypt them before running `kubectl
apply`. I eventually want to use Argo CD, though, so that prerequisite
step would make that a lot more complex. Eventually, I discovered
[KSOPS] and *Sealed Secrets*. KSOPS operates entirely on the client
side, and thus requires a plugin for Kustomize and/or Argo CD in order
to work, so it's not significantly different than the GnuPG/Make idea.
I like that Sealed Secrets does not require anything on the client side,
except when initially creating the manifests for the SealedSecret
objects, so Argo CD will "just work" without any extra tools or
configuration.
[Sealed Secrets]: https://github.com/bitnami-labs/sealed-secrets
[KSOPS]: https://github.com/viaduct-ai/kustomize-sops
The other day, when I was dealing with the mess that I accidentally
created by letting the *phpipam* MySQL database automaticall upgrade
itself, I attempted to restore from a Longhorn backup to try to get the
database working again. This did work, but as a side-effect, it changed
the storage class name of the *phpipam-pvc* persistent volume claim from
`longhorn` to `longhorn-static`. Now, when attempting to apply the
YAML manifest, `kubectl` complains because this field is immutable. As
such, the manifest needs to be updated to reflect the value set by
Longhorn when the backup was restored and the PVC was recreated.
The *fuse-device-plugin* handles mapping the `/dev/fuse` device into
unprivileged containers, e.g. for `buildah`.
Although *fuse-device-plugin* was recommended by Red Hat in their
blog post [How to use Podman inside of Kubernetes][0], it's probably
not the best choice any more. It's working for now, giving me the
ability to build container images in Kubernetes without running
`buildah` in a privileged container, but I will probably investigate
replacing it with the [generic-device-plugin][1] eventually.
[0]: https://www.redhat.com/sysadmin/podman-inside-kubernetes
[1]: https://github.com/squat/generic-device-plugin
The *dch-webhooks* tool now provides an operation for hosts to request a
signed SSH certificate from the SSH CA. It's primarily useful for
unattended deployments like CoreOS Ignition, where hosts do not have
any credentials to authenticate with the CA directly.
[Step CA] is an open-source online X.509 and SSH certificate authority
service. It supports issuing certificates via various protocols,
including ACME and its own HTTP API via the `step` command-line utility.
Clients can authenticate using a variety of methods, such as JWK, Open
ID Connect, or mTLS. This makes it very flexible and easy to introduce
to an existing ecosystem.
Although the CA service is mostly stateless, it does have an on-disk
database where stores some information, notably the list of SSH hosts
for which it has signed certificates. Most other operations, though, do
not require any persistent state; the service does not keep track of
every single certificate it signed, for example. It can be configured
to store authentication information (referred to as "provisioners") in
the database instead of the configuration file, by enabling the "remote
provisioner management" feature. This has the advantage of being able
to modify authentication configuration without updating a Kubernetes
ConfigMap and restarting the service.
The official Step CA documentation recommends using the `step ca init`
command initialize a new certificate authority. This command performs a
few steps:
* Generates an ECDSA key pair and uses it to create a self-signed root
certificate
* Generates a second ECDSA key pair and signs an intermediate CA
certificate using the root CA key
* Generates an ECDSA key pair and SSH root certificate
* Creates a `ca.json` configuration file
These steps can be performed separately, and in fact, I created the
intermediate CA certificate and signed it with the (offline) *dch Root
CA* certificate.
When the service starts for the first time, because
`authority/enableAdmin` is `true` and `authority/provisioners` is empty,
a new "Admin JWK" provisioner will be created automatically. This key
will be encrypted with the same password used to encrypt the
intermediate CA certificate private key, and can be used to create other
provisioners.
[Step CA]: https://smallstep.com/docs/step-ca/
phpIPAM supports "Apache authentication" which effectively delegates
authentication to the web server and trusts the `PHP_AUTH_USER` server
variable. This variable is usually set by an Apache authentication
module, but it can be set manually in the config. Here, we're using
`SetEnvIf` to populate it from the value of the `Remote-User` header
set by Authelia.
Using the *latest* tag for MariaDB is particularly problematic, as a
new version of the container may be pulled when the pod is scheduled on
a different host. MariaDB will not start in this case, as it recognizes
that the data on disk need to be upgraded.
To prevent database outages in situations like this, we need to pin to a
specific version of MariaDB, ensuring that every pod runs the same
version.
Having the Z-Wave and Zigbee admin interfaces exposed as sub-paths under
*homeassistant.pyrocufflink.blue* made it difficult to use Authelia.
Since I have a Firefox container tab specifically for Home Assistant,
the login redirect would open a new tab in a different container, since
Authelia is hosted at *auth.pyrocufflink.blue*. In order to log in, I
would have to temporarily disable "designated sites only" for the Home
Assistant tab container. Using subdomains for the admin interfaces
avoids this issue, since I can use a different container for them, one
that does not have the "designated sites only" setting, since I am less
worried about accidentally leaking data to sites on the Internet from
them.
Piper is the new text-to-speech service for Home Assistant. Whisper is
a speech-to-text service. Together, these services, which communicate
with Home Assistant via the Wyoming protocol, provide the speech
interface to the new Home Assistant Voice Assistant feature.
This commit adds resources for deploying the Home Assistant ecosystem
inside Kubernetes. Home Assistant itself, as well as Mosquitto, are
just normal Pods, managed by StatefulSets, that can run anywhere.
ZWaveJS2MQTT and Zigbee2MQTT, on the other hand, have to run on a
special node (a Raspberry Pi), where the respective controllers are
attached.
The Home Assistant UI is exposed externally via an Ingress resource.
The MQTT broker is also exposed externally, using the TCP proxy feature
of *ingress-nginx*. Additionally, the Zigbee2MQTT and ZWaveJS2MQTT
control panels are exposed via Ingress resources, but these are
protected by Authelia.
Hatch Learning Center has several domains; Tabitha couldn't decide which
she liked best :) At present, they all resolve to the same website, with
_hatchlearningcenter.org_ as the canonical name.
The *dch-webhooks* service is a generic tool I've written to handle
various automation flows. For now, it only has one feature: when a
transaction is created in Firefly-III, it searches Paperless-ngx for a
matching receipt, and if found, attaches it to the transaction.
Dustin