The last step in replacing Loki with Victoria Logs is to ingest logs
from Kubernetes pods. Like Promtail, Fluent Bit is capable of
augmenting log records with Kubernetes metadata, so we can search for
logs by pod name, namespace, etc. This of course requires access to the
Kubernetes API, and the easiest way to provide that is to run Fluent Bit
as a Kubernetes pod, granting its service account the appropriate
permissions.
Since Fluent Bit also collects logs from the systemd journal, I want to
make sure the configuration for that function stays the same on
Kubernetes nodes as on all other servers. One way to do that would be
to run two different instances of Fluent Bit: one managed by Ansible
that collects journal messages, and another managed by Kubernetes that
collects pod logs. This seems like unnecessary overhead, so I have
chosen a hybrid approach. Ansible manages the configuration for the
process running in Kubernetes.
Fluent Bit supports including configuration fragments from other files
using its `includes` option. Adding a glob pattern to the default
configuration will allow other roles to supply additional configuration
by creating files in the `/etc/fluent-bit/include` directory. This
makes composition of configuration significantly easier.
Unfortunately, `fluent-bit` has a quirk in that there must exist at
least one file matching the glob pattern, or it will fail to start. To
work around this, we must supply an empty fragment.
The _network.target_ unit should be used for ordering only. Listing it
as a `Requires=` dependency can cause _fluent-bit.service_ to fail to
start at all if the network takes slightly too long to initialize at
boot.
Fluent-bit does not have any native capability for setting a field with
the hostname of the machine, but it can set a field with the value of an
environment variable. Thus, we can set the `HOSTNAME` environment
variable and then use that to set the field in the pipeline.
When ingesting logs from sources other than systemd, such as
unstructured log files written by uncooperative services, it may be
necessary to define custom parsers.
[fluent-bit][0] is a generic, highly-configurable log collector. It was
apparently initially developed for fluentd, but is has so many output
capabilities that it works wil many different log aggregation systems,
including Victoria Logs.
Although Victoria Logs supports the Loki input format, and therefore
_Promtail_ would work, I want to try to avoid depending on third-party
repositories. _fluent-bit_ is packaged by Fedora, so there shouldn't be
any dependency issues, etc.
[0]: https://fluentbit.io
