Dustin C. Hatch
40e55a984b
All checks were successful
dustin/k8s-reboot-coordinator/pipeline/head This commit looks good
After some initial testing, I decided that the HTTP API approach to managing the reboot lock is not going to work. I originally implemented it this way so that the reboot process on the nodes could stay the same as it had always been, only adding a systemd unit to interact with the server to obtain the lock and drain the node. Unfortunately, this does not actually work in practice because there is no way to ensure that the new unit runs _first_ during the shutdown process. In fact, systemd practically _insists_ on stopping all running containers before any other units. The only solution, therefore, is to obtain the reboot lock and drain the node before initiating the actual shutdown procedure. I briefly considered installing a script on each node to handle all of this, and configuring _dnf-automatic_ to run that. I decided against that, though, as I would prefer to have as much of the node configuration managed by Kubnernetes as possible; I don't want to have to maintain that script with Ansible. I decided that the best way to resolve these issues was to rewrite the coordinator as a daemon that runs on every node. It waits for a sentinel file to appear (`/run/reboot-needed` by default), and then tries to obtain the reboot lock, drain the node, and reboot the machine. All of the logic is contained in the daemon and deployed by Kubernetes; the only change that has to be deployed by Ansible is configuring _dnf-automatic_ to run `touch /run/reboot-needed` instead of `shutdown -r +5`. This implementation is heavily inspired by [kured](https://kured.dev). Both rely on a sentinel file to trigger the reboot, but Kured uses a naive polling method for detecting it, which either means wasting a lot of CPU checking frequently, or introducing large delays by checking infrequently. Kured also implements the reboot lock without using a Lease, which may or may not be problematic if multiple nodes try to reboot simultaneously.
23 lines
461 B
Docker
23 lines
461 B
Docker
FROM docker.io/library/rust:1.88-alpine AS build
|
|
|
|
RUN apk --no-cache add musl-dev
|
|
|
|
WORKDIR /src
|
|
|
|
COPY . .
|
|
|
|
RUN cargo build --release --locked
|
|
|
|
RUN strip target/release/k8s-reboot-coordinator
|
|
|
|
|
|
FROM docker.io/library/busybox
|
|
|
|
COPY --from=build /src/target/release/k8s-reboot-coordinator /
|
|
|
|
ENV REBOOT_SENTINEL=/host/run/reboot-needed
|
|
|
|
ENTRYPOINT ["/k8s-reboot-coordinator"]
|
|
|
|
CMD ["nsenter", "-t", "1", "-m", "-n", "/bin/systemctl", "reboot", "--when", "+5 min"]
|