When Victoria Logs shuts down, it waits several seconds for clients to
disconnect themselves, then forcibly closes the connections to those
that do not. With the long-running streaming connection, there is no
way for the server to indicate that the client should disconnect itself.
This can cause data loss as messages that are sent during this window
are not consumed by Victoria Logs.
To improve the resilience of the sender, it now uses multiple short
streaming HTTP requests, instead of one long-running request. The relay
forwards messages from the MQTT subscriber to the sender's stream, then
closes the stream as soon as there is a 100ms delay between messages.
When the stream closes, the sender completes the HTTP request and
proceeds to the next iteration of the loop. The relay will not create a
new stream until a new message arrives from the MQTT subscriber.
With this approach, when Victoria Logs starts shutting down, there is a
significantly reduced opportunity for data loss. It is still possible
that messages sent in the request could be lost, if Victoria Logs
accepts the preflight request but not the actual stream. Addressing
this possibility would be quite a bit more complex, so hopefully it does
not become too much of a problem.
Dustin
3de1b53acd