Instead of "relaying" messages from the MQTT subscriber to the HTTP
request via a second channel, we now collect each batch of messages and
serialize them into a buffer. This makes it possible to retry the HTTP
request if it fails, without losing any data. Using the `Bytes` data
structure is the most effecient way to do this, as it implements `Clone`
without copying, so each iteration of the retry loop uses the same data
in memory.
Being able to retry failed HTTP requests eliminates the need for the
"preflight" request entirely.
When Victoria Logs shuts down, it waits several seconds for clients to
disconnect themselves, then forcibly closes the connections to those
that do not. With the long-running streaming connection, there is no
way for the server to indicate that the client should disconnect itself.
This can cause data loss as messages that are sent during this window
are not consumed by Victoria Logs.
To improve the resilience of the sender, it now uses multiple short
streaming HTTP requests, instead of one long-running request. The relay
forwards messages from the MQTT subscriber to the sender's stream, then
closes the stream as soon as there is a 100ms delay between messages.
When the stream closes, the sender completes the HTTP request and
proceeds to the next iteration of the loop. The relay will not create a
new stream until a new message arrives from the MQTT subscriber.
With this approach, when Victoria Logs starts shutting down, there is a
significantly reduced opportunity for data loss. It is still possible
that messages sent in the request could be lost, if Victoria Logs
accepts the preflight request but not the actual stream. Addressing
this possibility would be quite a bit more complex, so hopefully it does
not become too much of a problem.
The `metrics` crate, along with `metrics-exporter-prometheus`, makes it
extremely easy to instrument code with metrics and export them to
Prometheus. Here, we're adding a few metrics to track how many messages
are being processed and monitor the queue depth.
Dustin