When an instance is terminated, any bootstrap tokens assigned to it are
now deleted. Though these would expire anyway, deleting them ensures
that they cannot be used again if they happened to be leaked while the
instance was running. Further, it ensures that attempting to fetch the
`kubeadm` configuration for the instance will return an HTTP 404 Not
Found response once the instance has terminated.
The *GET /kubeadm/kubeconfig/<instance-id>* operation returns a
configuration document for `kubeadm` to add the node to the cluster as a
worker. The document is derived from the kubeconfig stored in the
`cluster-info` ConfigMap, which includes the external URL of the
Kubernetes API server and the root CA certificate used in the cluster.
The bootstrap token assigned to the specified instance is added to the
document for `kubeadm` to use for authentication. The kubeconfig is
stored in the ConfigMap as a string, so extracting data from it requires
deserializing the YAML document first.
In order to access the cluster information ConfigMap, the service
account bound to the pod running the provisioner service must have the
appropriate permissions.
The * GET /wireguard/config/<instance-id>* resource returns the
WireGuard client configuration assigned to the specified instance ID.
The resource contents are stored in the Kubernetes Secret, in a data
field named `wireguard-config`. The contents of this field are returned
directly as a string, without any transformation. Thus, the value must
be a complete, valid WireGuard configuration document. Instances will
fetch and save this configuration when they first launch, to configure
their access to the VPN.
Setting up the WireGuard client requires several pieces of information,
beyond the node private key and peer's public key. The peer endpoint
address/port, peer public key, and node IP address are also required.
As such, naming the resource a "key" is somewhat misleading.
In order to join the on-premises Kubernetes cluster, EC2 instances will
need to first connect to the WireGuard VPN. The *dynk8s* provisioner
will provide keys to instances to configure their WireGuard clients.
WireGuard keys must be pre-configured on the server and stored in
Kubernetes as *dynk8s.du5t1n.me/wireguard-key* Secret resources. They
must also have a `dynk8s.du5t1n.me/ec2-instance-id` label. If this
label is empty, the key is available to be assigned to an instance.
When an EventBridge event is received indicating an instance is now
running, a WireGuard key is assigned to that instance (by setting the
`dynk8s.du5t1n.me/ec2-instance-id` label). Conversely, when an event is
received indicating that the instance is terminated, any WireGuard keys
assigned to that instance are freed.
The lifecycle of ephemeral Kubernetes worker nodes is driven by events
emitted by Amazon EventBridge and delivered via Amazon Simple
Notification Service. These events trigger the *dynk8s* provisioner to
take the appropriate action based on the state of an EC2 instance.
In order to add a node to the cluster using `kubeadm`, a "bootstrap
token" needs to be created. When manually adding a node, this would be
done e.g. using `kubeadm token create`. Since bootstrap tokens are just
a special type of Secret, they can be easily created programmatically as
well. When a new EC2 instance enters the "running" state, the
provisioner creates a new bootstrap token and associates it with the
instance by storing the instance ID in a label in the Secret resource's
metadata.
The initial implementation of the event handler is rather naïve. It
generates a token for every instance, though some instances may not be
intended to be used as Kubernetes workers. Ideally, the provisioner
would only allocate tokens for instances matching some configurable
criteria, such as AWS tags. Further, a token is allocated every time
the instance enters the running state, even if a token already exists or
is not needed.
Dustin