victoria-metrics: Send alerts via ntfy

I don't like having alerts sent by e-mail.  Since I don't get e-mail
notifications on my watch, I often do not see alerts for quite some
time.  They are also much harder to read in an e-mail client (Fastmail
web an K-9 Mail both display them poorly).  I would much rather have
them delivered via _ntfy_, just like all the rest of the ephemeral
notifications I receive.

Fortunately, it is easy enough to integrate Alertmanager and _ntfy_
using the webhook notifier in Alertmanager.  Since _ntfy_ does not
natively support the Alertmanager webhook API, though, a bridge is
necessary to translate from one data format to the other.  There are a
few options for this bridge, but I chose
[alexbakker/alertmanager-ntfy][0] because it looked the most complete
while also having the simplest configuration format.  Sadly, it does not
expose any Prometheus metrics itself, and since it's deployed in the
_victoria-metrics_ namespace, it needs to be explicitly excluded from
the VMAgent scrape configuration.

[0]: https://github.com/alexbakker/alertmanager-ntfy
This commit is contained in:
2024-05-10 10:27:58 -05:00
parent a4591950ba
commit d74e26d527
5 changed files with 106 additions and 9 deletions

View File

@@ -0,0 +1,24 @@
http:
addr: :8000
ntfy:
baseurl: http://ntfy.ntfy:2586
notification:
topic: alerts
priority: |
status == "firing" ? "urgent" : "default"
tags:
- tag: heavy_check_mark
condition: status == "resolved"
- tag: rotating_light
condition: status == "firing"
templates:
title: >-
{{ if eq .Status "resolved" }}Resolved: {{ end -}}
{{ if index .Annotations "summary" -}}
{{ index .Annotations "summary" }}
{{ else -}}
{{ index .Labels "alertname" }}
{{ index .Labels "alertgroup" }}
{{ end -}}
description: |
{{ index .Annotations "description" }}