Authelia made breaking changes to the OIDC issuer configuration in 4.39,
specifically around what claims are present in identity tokens. Without
a claims policy set, clients will _not_ get the correct claims, which
breaks authentication and authorization in many cases (including
Kubernetes).
While I was fixing that, I went ahead and fixed a few of the other
deprecation warnings. There are still two that show up at startup, but
fixing them will be a bit more involved, it seems.
[ARA Records Ansible][0] is a results storage system for Ansible. It
provides a convenient UI for tracking Ansible playbooks and tasks. The
data are populated by an Ansible callback plugin.
ARA is a fairly simple Python+Django application. It needs a database
to store Ansible results, so we've connected it to the main PostgreSQL
database and configured it to connect and authenticate using mTLS.
Rather than mess with managing and distributing a static password for
ARA clients, I've configured Autheliad to allow anonymous access to
post data to the ARA API from within the private network or the
Kubernetes cluster. Access to the web UI does require authentication.
[0]: https://ara.recordsansible.org/
If there is an issue with the in-cluster database server, accessing the
Kubernetes API becomes impossible by normal means. This is because the
Kubernetes API uses Authelia for authentication and authorization, and
Authelia relies on the in-cluster database server. To solve this
chicken-and-egg scenario, I've set up a dedicated PostgreSQL database
server on a virtual machine, totally external to the Kubernetes cluster.
With this commit, I have changed the Authelia configuration to point at
this new database server. The contents of the new database server were
restored from a backup from the in-cluster server, so of Authelia's
state was migrated automatically. Thus, updating the configuration is
all that is necessary to switch to using it.
The new server uses certificate-based authentication. In order for
Authelia to access it, it needs a certificate issued by the
_postgresql-ca_ ClusterIssuer, managed by _cert-manager_. Although the
environment variables for pointing to the certificate and private key
are not listed explicitly in the Authelia documentation, their names
can be inferred from the configuration document schema and work as
expected.
The *hlcforms* application handles form submissions for the Hatch
Learning Center website. It has various features for Tabitha that are
only accessible internally, but the form submission handler itself of
course needs to be accessible anonymously.
A recent version of *Authelia* added a dark theme. Setting the `theme`
option to `auto` enables it when the user agent has the "prefers dark
mode" hint enabled.
Sometimes, I want to be able to look at active alerts without logging
in. This rule allows read-only access to the AlertManager UI and API.
Unfortunately, the user experience when attempting to create a new
Silence using the UI without first logging in is suboptimal, but I think
that's worth the trade-off.
After considering the implications of Authelia's pre-configured consent
feature, I decided I did not like the fact that a malicious program
could potentially take over my entire Kubernetes cluster without my
knowledge, since `kubectl` may not require any interaction, and could
therefore be executed without my knowledge. I stopped ticking the
"Remember Consent" checkbox out of paranoia, but that's gotten kind of
annoying. I figure a good compromise is to only prompt for consent a
couple of times per day.
Since we've configured the Ingress for Firefly III to log everyone in as
*dustin* via a faked `Remote-User` request header, any user on the
Pyrocufflink domain would be able to see my finances. Using Authelia's
access control mechanism, we can restrict this to only users in a
specific group.
By default, Authelia uses a local SQLite database for persistent data
(e.g. authenticator keys, TOTP secrets, etc.) and keeps session data in
memory. Together, these have some undesirable side effects. First,
since needing access to the filesystem to store the SQLite database
means that the pod has to be managed by a StatefulSet. Restarting
StatefulSet pods means stopping them all and then starting them back up,
which causes downtime. Additionally, the SQLite database file needs to
be backed up, which I never got around to setting up. Further, any time
the service is restarted, all sessions are invalidated, so users have to
sign back in.
All of these issues can be resolved by configuring Authelia to store all
of its state externally. The persistent data can be stored in a
PostgreSQL database and the session state can be stored in Redis. Using
a database managed by the existing Postgres Operator infrastructure
automaticaly enables high availability and backups as well.
To migrate the contents of the database, I used [pgloader]. With
Authelia shut down, I ran the migration job. Authelia's database schema
is pretty simple, so there were no problems with the conversion.
Authelia started back up with the new database configuration without any
issues.
Session state are still stored only in memory of the Redis process.
This is probably fine, since Redis will not need restarted often, except
for updates. At least restarting Authelia to adjust its configuration
will not log everyone out.
[pgloader]: https://pgloader.readthedocs.io/en/latest/ref/sqlite.html
The `argocd` command needs to have its own OIDC client configuration,
since it works like a "public" client. To log in, run
```sh
argocd login argocd.pyrocufflink.blue --sso
```
[Argo CD] is a Kubernetes-native GitOps/continuous deployment manager.
It monitors the state of Kubnernetes resources, such as Pods,
Deployments, ConfigMaps, Secrets, and Custom Resources, and synchronizes
them with their canonical definitions from a Git repository.
*Argo CD* consists of various components, including a Repository
Service, an Application Controller, a Notification Controller, and an
API server/Web UI. It also has some optional components, such as a
bundled Dex server for authentication/authorization, and an
ApplicationSet controller, which we will not be using.
[Argo CD]: https://argo-cd.readthedocs.io/
[Firefly III][0] is a free and open source, web-based personal finance
management application. It features a double-entry bookkeeping system
for tracking transactions, plus other classification options like
budgets, categories, and tags. It has a rule engine that can
automatically manipulate transactions, plus several other really useful
features.
The application itself is mostly standard browser-based GUI written in
PHP. There is an official container image, though it is not
particularly well designed and must be run as root (it does drop
privileges before launching the actual application, thankfully). I may
decide to create a better image later.
Along with the main application, there is a separate tool for importing
transactions from a CSV file. Its design is rather interesting: though
it is a web-based application, it does not have any authentication or
user management, but uses a user API key to access the main Firefly III
application. This effectively requires us to have one instance of the
importer per user. While not ideal, it isn't particularly problematic
since there are only two of us (and Tabitha may not even end up using
it; she seems to like YNAB).
[0]: https://www.firefly-iii.org/
This configuration is for the instance of MinIO running on the BURP
server, which will be used to store PostgreSQL backups created by the
Postgres Operator.
By default, Authelia requires the user to explicitly consent to allow
an application access to personal information *every time the user
authenticates*. This is rather annoying, so luckily, it provides a
way to remember the consent for a period of time.
For convenience, clients on the internal network do not need to
authenticate in order to access *scanserv-js*. There isn't anything
particularly sensitive about this application, anyway.
Enabling OpenID Connect authentication for the Kubernetes API server
will allow clients, particularly `kubectl` to log in without needing
TLS certificates and private keys.
Authelia can act as an Open ID Connect identity provider. This allows
it to provide authentication/authorization for other applications
besides those inside the Kubernetes cluster using it for Ingress
authentication.
To start with, we'll configure an OIDC client for Jenkins.
Authelia is a general authentication provider that works (primarily)
by integrating with *nginx* using its subrequest mechanism. It works
great with Kubernetes/*ingress-nginx* to provide authentication for
services running in the cluster, especially those that do not provide
their own authentication system.
Authelia needs a database to store session data. It supports various
engines, but since we're only running a very small instance with no real
need for HA, SQLite on a Longhorn persistent volume is sufficient.
Configuration is done mostly through a YAML document, although some
secret values are stored in separate files, which are pointed to by
environment variables.
Dustin