Whenever a lock request is made for a host that is a node in the current
Kubernetes cluster, the node will now be cordoned and all pods evicted
from it. The HTTP request will not return until all pods are gone,
making the lock request suitable for use in a system shutdown step.
This commit introduces two HTTP path operations:
* POST /api/v1/lock: Acquire a reboot lock
* POST /api/v1/unlock: Release a reboot lock
Both operations take a _multipart/form-data_ or
_application/x-www-form-urlencoded_ body with a required `hostname`
field. This field indicates the name of the host acquiring/releasing
the lock. the `lock` operation also takes an optional `wait` field. If
this value is provided with a `false` value, and the reboot lock cannot
be acquired immediately, the request will fail with an HTTP 419
conflict. If a `true` value is provided, or the field is omitted, the
request will block until the lock can be acquired.
Locking is implemented with a Kubernetes Lease resource using
Server-Side Apply. By setting the field manager of the `holderIdentity`
field to match its value, we can ensure that there are no race
conditions in acquiring the lock; Kubernetes will reject the update if
both the new value and the field manager do not match. This is
significantly safer than a more naïve check-then-set approach.
Since the API provided by this service is intended to be used on the
command line e.g. with `curl`, we need our responses to have a trailing
newline. This ensures that, when used interactively, the next shell
prompt is correctly placed on a new line, and when used
non-interactively, line-buffered output is correctly flushed (i.e. to a
log file).
Dustin