The [Kubernetes Credentials Provider][0] plugin for Jenkins allows
Jenkins to expose Kubernetes Secret resources as Jenkins Credentials.
Jobs can use them like normal Jenkins credentials, e.g. using
`withCredentials`, `sshagent`, etc. The only drawback is that every
credential exposed this way is available to every job, at least until
[PR #40][1] is merged. Fortunately, jobs managed by this Jenkins
instance are all trusted; no anonymous pull requests are possible, so
the risk is mitigated.
[0]: https://jenkinsci.github.io/kubernetes-credentials-provider-plugin/
[1]: https://github.com/jenkinsci/kubernetes-credentials-provider-plugin/pull/40
Setting the `imagePullSecrets` property on the default service account
for the *jenkins-jobs* namespace allows jobs to run from private
container images automatically, without additional configuration in the
pipeline definitions.
[sshca] is a simple web service I wrote to automatically create signed
SSH certificates for hosts' public keys. It authenticates hosts by
their machine UUID, which it can find using the libvirt API.
[sshca]: https://git.pyrocufflink.net/dustin/sshca
The Raspberry Pi usually has the most free RAM of all the Kubernetes
nodes, so pods tend to get assigned there even when it would not be
appropriate. Jenkins, for example definitely does not need to run
there, so let's force it to run on the bigger nodes.
Argo CD will delete and re-create this Job each time it synchronizes the
*jenkins* application. The job creates a snapshot of the Jenkins volume
using an HTTP request to the Longhorn UI.
When migrating the `pod-secrets` Secret to a SealedSecret, I
accidentally created it using the `--from-file` instead of
`--from-env-file` argument to `kubectl secret create generic`. This had
the effect of creating a single key named `pod.secrets` with the entire
contents of the file as its value. This broke backups to MinIO, since
the PostgreSQL containers could no longer read the credentials from the
environment. Regenerating the SealedSecret with the correct arguments
resolves this issue.
By default, Authelia uses a local SQLite database for persistent data
(e.g. authenticator keys, TOTP secrets, etc.) and keeps session data in
memory. Together, these have some undesirable side effects. First,
since needing access to the filesystem to store the SQLite database
means that the pod has to be managed by a StatefulSet. Restarting
StatefulSet pods means stopping them all and then starting them back up,
which causes downtime. Additionally, the SQLite database file needs to
be backed up, which I never got around to setting up. Further, any time
the service is restarted, all sessions are invalidated, so users have to
sign back in.
All of these issues can be resolved by configuring Authelia to store all
of its state externally. The persistent data can be stored in a
PostgreSQL database and the session state can be stored in Redis. Using
a database managed by the existing Postgres Operator infrastructure
automaticaly enables high availability and backups as well.
To migrate the contents of the database, I used [pgloader]. With
Authelia shut down, I ran the migration job. Authelia's database schema
is pretty simple, so there were no problems with the conversion.
Authelia started back up with the new database configuration without any
issues.
Session state are still stored only in memory of the Redis process.
This is probably fine, since Redis will not need restarted often, except
for updates. At least restarting Authelia to adjust its configuration
will not log everyone out.
[pgloader]: https://pgloader.readthedocs.io/en/latest/ref/sqlite.html
The PostgreSQL server managed by *Postgres Operator* uses a self-signed
certificate by default. In order to enable full validation of the
server certificate, we need to use a certificate signed by a known CA
that the clients can trust. To that end, I have added a *cert-manager*
Issuer specifically for PostgreSQL. The CA certificate is also managed
by *cert-manager*; it is self-signed and needs to be distributed to
clients out-of-band.
The `config.yml` document for *kitchen* contains several "secret" values
(e.g. passwords to Nextcloud, MQTT, etc.). We don't want to commit
these to the Git repository, of course, but as long as Kustomize expects
to find the `config.yml` file, we won't be able to manage the
application with Argo CD. Ultimately, *kitchen* needs to be modified to
be able to read secrets separately from config, but until then, we will
have to avoid managing `config.yml` with Kustomize.
I actually created this a long time ago, but forgot to update the
manifest in Git.
The *homeassistant* database is used by Home Assistant for its
*recorder* component, which stores long-term statistics. The data
stored here are only used for e.g. History and Logbook; current entity
states are still stored on the filesystem.
The `argocd` command needs to have its own OIDC client configuration,
since it works like a "public" client. To log in, run
```sh
argocd login argocd.pyrocufflink.blue --sso
```
Without `disableNameSuffixHash` enabled, Kustomize will create a unique
ConfigMap any time the contents of source file change. It will also
update any Deployment, StatefulSet, etc resources to point to the new
ConfigMap. This has the effect of restarting any pods that refer to the
ConfigMap whenever its contents change.
I had avoided using this initially because Kustomize does *not* delete
previous ConfigMap resources whenever it creates a new one. Now that we
have Argo CD, though, this is not an issue, as it will clean up the old
resources whenever it synchronizes.
[Argo CD] is a Kubernetes-native GitOps/continuous deployment manager.
It monitors the state of Kubnernetes resources, such as Pods,
Deployments, ConfigMaps, Secrets, and Custom Resources, and synchronizes
them with their canonical definitions from a Git repository.
*Argo CD* consists of various components, including a Repository
Service, an Application Controller, a Notification Controller, and an
API server/Web UI. It also has some optional components, such as a
bundled Dex server for authentication/authorization, and an
ApplicationSet controller, which we will not be using.
[Argo CD]: https://argo-cd.readthedocs.io/
[Sealed Secrets] will allow us to store secret values in the Git
repository, since the actual secrets are encrypted and can only be
decrypted using the private key stored in the Kubernetes cluster.
I have been looking for a better way to deal with secrets for some time
now. For one thing, having the secret files ignored by Git means they
only exist on my main desktop. If I need to make changes to an
application from another machine, I have to not only clone the
repository, but also manually copy the secret files. That sort of
makes my desktop a single point-of-failure. I tried moving all the
secret files to another (private) repository and adding it as a
submodule, but Kustomize did not like that; it will only load files from
the current working directory, or another Kustomize project. Having to
create two projects for each application, one for the secrets and one
for everything else, would be tedious and annoying. I also considered
encrypting all the secret files with e.g. GnuPG and creating Make
recipies for each project to decrypt them before running `kubectl
apply`. I eventually want to use Argo CD, though, so that prerequisite
step would make that a lot more complex. Eventually, I discovered
[KSOPS] and *Sealed Secrets*. KSOPS operates entirely on the client
side, and thus requires a plugin for Kustomize and/or Argo CD in order
to work, so it's not significantly different than the GnuPG/Make idea.
I like that Sealed Secrets does not require anything on the client side,
except when initially creating the manifests for the SealedSecret
objects, so Argo CD will "just work" without any extra tools or
configuration.
[Sealed Secrets]: https://github.com/bitnami-labs/sealed-secrets
[KSOPS]: https://github.com/viaduct-ai/kustomize-sops
The other day, when I was dealing with the mess that I accidentally
created by letting the *phpipam* MySQL database automaticall upgrade
itself, I attempted to restore from a Longhorn backup to try to get the
database working again. This did work, but as a side-effect, it changed
the storage class name of the *phpipam-pvc* persistent volume claim from
`longhorn` to `longhorn-static`. Now, when attempting to apply the
YAML manifest, `kubectl` complains because this field is immutable. As
such, the manifest needs to be updated to reflect the value set by
Longhorn when the backup was restored and the PVC was recreated.
The *fuse-device-plugin* handles mapping the `/dev/fuse` device into
unprivileged containers, e.g. for `buildah`.
Although *fuse-device-plugin* was recommended by Red Hat in their
blog post [How to use Podman inside of Kubernetes][0], it's probably
not the best choice any more. It's working for now, giving me the
ability to build container images in Kubernetes without running
`buildah` in a privileged container, but I will probably investigate
replacing it with the [generic-device-plugin][1] eventually.
[0]: https://www.redhat.com/sysadmin/podman-inside-kubernetes
[1]: https://github.com/squat/generic-device-plugin
The *dch-webhooks* tool now provides an operation for hosts to request a
signed SSH certificate from the SSH CA. It's primarily useful for
unattended deployments like CoreOS Ignition, where hosts do not have
any credentials to authenticate with the CA directly.
[Step CA] is an open-source online X.509 and SSH certificate authority
service. It supports issuing certificates via various protocols,
including ACME and its own HTTP API via the `step` command-line utility.
Clients can authenticate using a variety of methods, such as JWK, Open
ID Connect, or mTLS. This makes it very flexible and easy to introduce
to an existing ecosystem.
Although the CA service is mostly stateless, it does have an on-disk
database where stores some information, notably the list of SSH hosts
for which it has signed certificates. Most other operations, though, do
not require any persistent state; the service does not keep track of
every single certificate it signed, for example. It can be configured
to store authentication information (referred to as "provisioners") in
the database instead of the configuration file, by enabling the "remote
provisioner management" feature. This has the advantage of being able
to modify authentication configuration without updating a Kubernetes
ConfigMap and restarting the service.
The official Step CA documentation recommends using the `step ca init`
command initialize a new certificate authority. This command performs a
few steps:
* Generates an ECDSA key pair and uses it to create a self-signed root
certificate
* Generates a second ECDSA key pair and signs an intermediate CA
certificate using the root CA key
* Generates an ECDSA key pair and SSH root certificate
* Creates a `ca.json` configuration file
These steps can be performed separately, and in fact, I created the
intermediate CA certificate and signed it with the (offline) *dch Root
CA* certificate.
When the service starts for the first time, because
`authority/enableAdmin` is `true` and `authority/provisioners` is empty,
a new "Admin JWK" provisioner will be created automatically. This key
will be encrypted with the same password used to encrypt the
intermediate CA certificate private key, and can be used to create other
provisioners.
[Step CA]: https://smallstep.com/docs/step-ca/
phpIPAM supports "Apache authentication" which effectively delegates
authentication to the web server and trusts the `PHP_AUTH_USER` server
variable. This variable is usually set by an Apache authentication
module, but it can be set manually in the config. Here, we're using
`SetEnvIf` to populate it from the value of the `Remote-User` header
set by Authelia.
Using the *latest* tag for MariaDB is particularly problematic, as a
new version of the container may be pulled when the pod is scheduled on
a different host. MariaDB will not start in this case, as it recognizes
that the data on disk need to be upgraded.
To prevent database outages in situations like this, we need to pin to a
specific version of MariaDB, ensuring that every pod runs the same
version.
Dustin