The last step in replacing Loki with Victoria Logs is to ingest logs
from Kubernetes pods. Like Promtail, Fluent Bit is capable of
augmenting log records with Kubernetes metadata, so we can search for
logs by pod name, namespace, etc. This of course requires access to the
Kubernetes API, and the easiest way to provide that is to run Fluent Bit
as a Kubernetes pod, granting its service account the appropriate
permissions.
Since Fluent Bit also collects logs from the systemd journal, I want to
make sure the configuration for that function stays the same on
Kubernetes nodes as on all other servers. One way to do that would be
to run two different instances of Fluent Bit: one managed by Ansible
that collects journal messages, and another managed by Kubernetes that
collects pod logs. This seems like unnecessary overhead, so I have
chosen a hybrid approach. Ansible manages the configuration for the
process running in Kubernetes.
[fluent-bit][0] is a generic, highly-configurable log collector. It was
apparently initially developed for fluentd, but is has so many output
capabilities that it works wil many different log aggregation systems,
including Victoria Logs.
Although Victoria Logs supports the Loki input format, and therefore
_Promtail_ would work, I want to try to avoid depending on third-party
repositories. _fluent-bit_ is packaged by Fedora, so there shouldn't be
any dependency issues, etc.
[0]: https://fluentbit.io
