The _haproxy_ role only installs HAProxy and provides some basic global
configuration; it expects another role to depend on it and provide
concrete proxy configuration with drop-in configuration files. Thus, we
need a role specifically for the Kubernetes control plane nodes to
provide the configuration to proxy for the API server.
Control plane nodes will now run _keepalived_, to provide a "floating"
IP address that is assigned to one of the nodes at a time. This
address (172.30.0.169) is now the target of the DNS A record for
_kubernetes.pyrocufflink.blue_, so clients will always communicate with
the server that currently holds the floating address, whichever that may
be.
I was originally inspired by the official Kubernetes [High Availability
Considerations][0] document when designing this. At first, I planned to
deploy _keepalived_ and HAProxy as DaemonSets on the control plane
nodes, but this ended up being somewhat problematic whenever all of the
control plane nodes would go down at once, as the _keepalived_ and
HAProxy pods would not get scheduled and thus no clients communicate
with the API servers.
[0]: 9d7cfab6fe/docs/ha-considerations.md
The `kubernetes.yml` playbook now applies the _kubelet_ role to hosts in
the _k8s-controller_ group. This will prepare them to join the cluster
as control plane nodes, but will not actually add them to the cluster.
So far, I have been managing Kubernetes worker nodes with Fedora CoreOS
Ignition, but I have decided to move everything back to Fedora and
Ansible. I like the idea of an immutable operating system, but the FCOS
implementation is not really what I want. I like the automated updates,
but that can be accomplished with _dnf-automatic_. I do _not_ like
giving up control of when to upgrade to the next Fedora release.
Mostly, I never did come up with a good way to manage application-level
configuration on FCOS machines. None of my experiments (Cue+tmpl,
KCL+etcd+Luci) were successful, which mostly resulted in my manually
managing configuration on nodes individually. Managing OS-level
configuration is also rather cumbersome, since it requires redeploying
the machine entirely. Altogether, I just don't think FCOS fits with my
model of managing systems.
This commit introduces a new playbook, `kubernetes.yml`, and a handful of
new roles to manage Kubernetes worker nodes running Fedora Linux. It
also adds two new deploy scripts, `k8s-worker.sh` and `k8s-longhorn.sh`,
which fully automate the process of bringing up worker nodes.
Dustin