v-m: Scrape metrics from Patroni

Patroni, a component of the *postgres poerator*, exports metrics about
the PostgreSQL database servers it manages.  Notably, it provides
information about the current transaction log location for each server.
This allows us to monitor and alert on the health of database replicas.

victoria-metrics/alerts.yml

@@ -126,3 +126,12 @@ groups:
   - alert: Battery Low
     expr:
       homeassistant_sensor_battery_percent{entity!~"sensor\\.(pixel_|sm_p610).*"} < 10
+
+- name: PostgreSQL
+  rules:
+  - alert: Replica lag too high
+    expr:
+      (patroni_xlog_location != 0)
+      - ignoring (instance) group_right (scope) (patroni_xlog_replayed_location != 0)
+      > 10240
+    for: 10m

victoria-metrics/scrape.yml

@@ -338,3 +338,24 @@ scrape_configs:
   static_configs:
   - targets:
     - argocd-metrics.argocd:8082
+
+- job_name: patroni
+  kubernetes_sd_configs:
+  - role: pod
+    namespaces:
+      names:
+      - postgresql
+    selectors:
+    - role: pod
+      label: application=spilo
+  relabel_configs:
+  - action: keep
+    source_labels:
+    - __meta_kubernetes_pod_container_name
+    regex: postgres
+  - action: keep
+    source_labels:
+    - __meta_kubernetes_pod_container_port_number
+    regex: '8008'
+  - source_labels: [__meta_kubernetes_pod_name]
+    target_label: instance