Dustin
d74e26d527
I don't like having alerts sent by e-mail. Since I don't get e-mail notifications on my watch, I often do not see alerts for quite some time. They are also much harder to read in an e-mail client (Fastmail web an K-9 Mail both display them poorly). I would much rather have them delivered via _ntfy_, just like all the rest of the ephemeral notifications I receive. Fortunately, it is easy enough to integrate Alertmanager and _ntfy_ using the webhook notifier in Alertmanager. Since _ntfy_ does not natively support the Alertmanager webhook API, though, a bridge is necessary to translate from one data format to the other. There are a few options for this bridge, but I chose [alexbakker/alertmanager-ntfy][0] because it looked the most complete while also having the simplest configuration format. Sadly, it does not expose any Prometheus metrics itself, and since it's deployed in the _victoria-metrics_ namespace, it needs to be explicitly excluded from the VMAgent scrape configuration. [0]: https://github.com/alexbakker/alertmanager-ntfy |
||
|---|---|---|
| .. | ||
| README.md | ||
| alertmanager-ntfy.config.yml | ||
| alertmanager-ntfy.yaml | ||
| alertmanager.config.yml | ||
| alertmanager.yaml | ||
| alerts.yml | ||
| blackbox-exporter.yaml | ||
| blackbox.yml | ||
| ingress.yaml | ||
| kustomization.yaml | ||
| namespace.yaml | ||
| scrape.yml | ||
| secrets.yaml | ||
| vmagent.yaml | ||
| vmalert.yaml | ||
| vminsert.yaml | ||
| vmselect.yaml | ||
| vmstorage.yaml | ||
README.md
Victoria Metrics
Victoria Metrics is a powerful, scalable time-series database compatible with Prometheus and its ecosystem of metrics exporters.
Clustered Deployment
Victoria Metrics can run in a high-availability cluster, with the various functions of the TSDB split into independently-scalable processes:
vmstorage: Stores time series data.vminsert: Ingests metrics in various formats (e.g. Prometheus) and sends them to one or morevmstoragenodes.vmselect: Performs metrics queries, retrieving results from one or morevmstoragenodes.
The vmstorage processes are managed by a StatefulSet with a volume claim
template for persistent storage. The number of replicas in the StatefulSet
must be 2n-1 where n is the value of the replicationFactor setting for
vminsert.
vminsert and vmselect processes are stateless and thus managed by a
Deployment. There should be at least 2 replicas of each of these, so that
restarts, etc. can be performed without any downtime.
vmagent
In a typical Victoria Metrics ecosystem, collecting metrics is handled
separately from the TSDB. The vmagent process handles scraping and receiving
metrics and passing them to vminsert. vmagent can cache received metrics
locally, in case no vminsert process is available, so it requires persistent
storage and is therefore managed by a StatefulSet. Because there are multiple
vmagent processes scraping the same targets, the vminsert and vmstorage
processes MUST have the dedup.minScrapeInterval setting set to match the
vmagent scrape interval. Jobs with scrape intervals longer than the
default will unfortunately have duplicate data points.
Blackbox Exporter
Many applications and web sites are monitored via the Blackbox Exporter, which makes arbitrary HTTP, TCP, ICMP, etc. requests and reports Prometheus metrics about them. This is a stateless process, managed by a Deployment.
vmalert
Victoria Metrics has a separate process for alerting, vmalert. This process periodically executes the queries defined in its alerting rules and creates alerts for matching results. Alerts are stored in the Victoria Metrics TSDB. Rules are defined in a YAML document, managed by a ConfigMap. Notifications are sent to Alertmanager.
Alertmanager
Alertmanager receives notifications from vmalert and sends e.g. email
messages. Multiple instances can be run in a cluster; each node needs to know
the host and port of every node in the cluster.