Sometimes, Grafana gets pretty slow, especially when it's running on one
of the Raspberry Pi nodes. When this happens, the health check may take
longer than the default timeout of 1 second to respond. This then marks
the pod as unhealthy, even though it's still working.
The legacy alerting feature (which we never used) has been deprecated
for a long time and removed in Grafana 11. The corresponding
configuration block must be removed from the config file or Grafana will
not start.
Usually, Grafana datastores are configured using its web GUI. When
setting up a datastore that requires TLS client authentication, the
client certificate and private key have to be pasted into the form.
For certificates that renew frequently, this method would require a
frequent manual effort. Fortunately, Grafana supports defining
datastores via its "provisioning" mechanism, reading the configuration
from YAML files on the filesystem.
Now that Victoria Metrics is hosted in Kubernetes, it only makes sense
to host Grafana there as well. I chose to use a single-instance
deployment for simplicity; I don't really need high availability for
Grafana. Its configuration does not change enough to worry about the
downtime associated with restarting it. Migrating the existing data
from SQLite to PostgreSQL, while possible, is just not worth the hassle.
