When migrating the `pod-secrets` Secret to a SealedSecret, I
accidentally created it using the `--from-file` instead of
`--from-env-file` argument to `kubectl secret create generic`. This had
the effect of creating a single key named `pod.secrets` with the entire
contents of the file as its value. This broke backups to MinIO, since
the PostgreSQL containers could no longer read the credentials from the
environment. Regenerating the SealedSecret with the correct arguments
resolves this issue.
By default, Authelia uses a local SQLite database for persistent data
(e.g. authenticator keys, TOTP secrets, etc.) and keeps session data in
memory. Together, these have some undesirable side effects. First,
since needing access to the filesystem to store the SQLite database
means that the pod has to be managed by a StatefulSet. Restarting
StatefulSet pods means stopping them all and then starting them back up,
which causes downtime. Additionally, the SQLite database file needs to
be backed up, which I never got around to setting up. Further, any time
the service is restarted, all sessions are invalidated, so users have to
sign back in.
All of these issues can be resolved by configuring Authelia to store all
of its state externally. The persistent data can be stored in a
PostgreSQL database and the session state can be stored in Redis. Using
a database managed by the existing Postgres Operator infrastructure
automaticaly enables high availability and backups as well.
To migrate the contents of the database, I used [pgloader]. With
Authelia shut down, I ran the migration job. Authelia's database schema
is pretty simple, so there were no problems with the conversion.
Authelia started back up with the new database configuration without any
issues.
Session state are still stored only in memory of the Redis process.
This is probably fine, since Redis will not need restarted often, except
for updates. At least restarting Authelia to adjust its configuration
will not log everyone out.
[pgloader]: https://pgloader.readthedocs.io/en/latest/ref/sqlite.html
The PostgreSQL server managed by *Postgres Operator* uses a self-signed
certificate by default. In order to enable full validation of the
server certificate, we need to use a certificate signed by a known CA
that the clients can trust. To that end, I have added a *cert-manager*
Issuer specifically for PostgreSQL. The CA certificate is also managed
by *cert-manager*; it is self-signed and needs to be distributed to
clients out-of-band.
I actually created this a long time ago, but forgot to update the
manifest in Git.
The *homeassistant* database is used by Home Assistant for its
*recorder* component, which stores long-term statistics. The data
stored here are only used for e.g. History and Logbook; current entity
states are still stored on the filesystem.
[Firefly III][0] is a free and open source, web-based personal finance
management application. It features a double-entry bookkeeping system
for tracking transactions, plus other classification options like
budgets, categories, and tags. It has a rule engine that can
automatically manipulate transactions, plus several other really useful
features.
The application itself is mostly standard browser-based GUI written in
PHP. There is an official container image, though it is not
particularly well designed and must be run as root (it does drop
privileges before launching the actual application, thankfully). I may
decide to create a better image later.
Along with the main application, there is a separate tool for importing
transactions from a CSV file. Its design is rather interesting: though
it is a web-based application, it does not have any authentication or
user management, but uses a user API key to access the main Firefly III
application. This effectively requires us to have one instance of the
importer per user. While not ideal, it isn't particularly problematic
since there are only two of us (and Tabitha may not even end up using
it; she seems to like YNAB).
[0]: https://www.firefly-iii.org/
While I was preparing to deploy PostgreSQL for Firefly III, I was
thinking it would be a neat idea to write an operator that uses
custom resources to manage PostgreSQL roles and databases. Then I
though, surely something like that must exist already. As it turns out,
the [Postgres Operator][0] does exactly that, and a whole lot more.
The *Postgres Operator* handles deploying PostgreSQL server instances,
including primary/standby replication with load balancers. It uses
custom resources to manage the databases and users (roles) in an
instance, and stores role passwords in Secret resources. It supports
backing up instances using `pg_basebackup` and WAL archives (i.e.
physical backups) via [WAL-E][1]/[WAL-G][2]. While various backup
storage targets are supported, *Postgres Operator* really only works
well with the cloud storage services like S3, Azure, and Google Cloud
Platform. Fortunately, S3-compatible on-premises solutions like MinIO
are just fine.
I think for my use cases, a single PostgreSQL cluster with multiple
databases will be sufficient. I know *Firefly III* will need a
PostgreSQL database, and I will likely want to migrate *Paperless-ngx*
to PostgreSQL eventually too. Having a single instance will save on
memory resources, at the cost of per-application point-in-time recovery.
For now, just one server in the cluster is probably sufficient, but
luckily adding standby servers appears to be really easy should the need
arise.
[0]: https://postgres-operator.readthedocs.io/en/latest/
[1]: https://github.com/wal-e/wal-e
[2]: https://github.com/wal-g/wal-g
Dustin