In order to join the on-premises Kubernetes cluster, EC2 instances will
need to first connect to the WireGuard VPN. The *dynk8s* provisioner
will provide keys to instances to configure their WireGuard clients.
WireGuard keys must be pre-configured on the server and stored in
Kubernetes as *dynk8s.du5t1n.me/wireguard-key* Secret resources. They
must also have a `dynk8s.du5t1n.me/ec2-instance-id` label. If this
label is empty, the key is available to be assigned to an instance.
When an EventBridge event is received indicating an instance is now
running, a WireGuard key is assigned to that instance (by setting the
`dynk8s.du5t1n.me/ec2-instance-id` label). Conversely, when an event is
received indicating that the instance is terminated, any WireGuard keys
assigned to that instance are freed.
The lifecycle of ephemeral Kubernetes worker nodes is driven by events
emitted by Amazon EventBridge and delivered via Amazon Simple
Notification Service. These events trigger the *dynk8s* provisioner to
take the appropriate action based on the state of an EC2 instance.
In order to add a node to the cluster using `kubeadm`, a "bootstrap
token" needs to be created. When manually adding a node, this would be
done e.g. using `kubeadm token create`. Since bootstrap tokens are just
a special type of Secret, they can be easily created programmatically as
well. When a new EC2 instance enters the "running" state, the
provisioner creates a new bootstrap token and associates it with the
instance by storing the instance ID in a label in the Secret resource's
metadata.
The initial implementation of the event handler is rather naïve. It
generates a token for every instance, though some instances may not be
intended to be used as Kubernetes workers. Ideally, the provisioner
would only allocate tokens for instances matching some configurable
criteria, such as AWS tags. Further, a token is allocated every time
the instance enters the running state, even if a token already exists or
is not needed.
The `terraform` directory contains the resource descriptions for all AWS
services that need to be configured in order for the dynamic K8s
provisioner to work. Specifically, it defines the EventBridge rule and
SNS topic/subscriptions that instruct AWS to send EC2 instance state
change notifications to the *dynk8s-provisioner*'s HTTP interface.
dustin/dynk8s-provisioner/pipeline/head This commit looks goodDetails
Fedora 36 has OpenSSL 3, while the *rust* container image has OpenSSL
1.1. Since Fedora 35 is still supported, and it includes OpenSSL 1.1,
we can use it as our base for the runtime image.
Upon receipt of a notification or unsubscribe confirmation message from
SNS, after the message signature has been verified, the receiver will
now write the re-serialized contents of the message out to the
filesystem. This will allow the messages to be inspected later in order
to develop additional functionality for this service.
The messages are saved in a `messages` director within the current
working directory. This directory contains a subdirectory for each SNS
topic. Within the topic subdirectories, the each message is saved in a
file named with the message timestamp and ID.
This commit introduces the HTTP interface for the dynamic K8s node
provisioner. It will serve as the main communication point between the
ephemeral nodes in the cloud, sharing the keys and tokens they require
in order to join the Kubernetes cluster.
The initial functionality is simply an Amazon SNS notification receiver.
SNS notifications will be used to manage the lifecycle of the dynamic
nodes.
For now, the notification receiver handles subscription confirmation
messages by following the link provided to confirm the subscription.
All other messages are simply written to the filesystem; these will be
used to implement and test future functionality.
The `model::sns::Message` enumeration provides a mechanism for
deserializing a JSON document into the correct type. It will be used by
the HTTP operation that receives messages from SNS in order to determine
the correct action to take in response to the message.
In order to prevent arbitrary clients from using the provisioner to
retrieve WireGuard keys and Kubernetes bootstrap tokens, access to those
resources *must* be restricted to the EC2 machines created by the
Kubernetes Cloud Autoscaler. The key to the authentication process will
be SNS notifications from AWS to indicate when new EC2 instances are
created; everything that the provisioner does will be associated with an
instance it discovered through an SNS notification.
SNS messages are signed using PKCS#1 v1.5 RSA-SHA1, with a public key
distributed in an X.509 certificate. To ensure that messages received
are indeed from AWS, the provisioner will need to verify those
signatures. Messages with missing or invalid signatures will be
considered unsafe and ignored.
The `model::sns` module includes the data structures that represent SNS
messages. The `sns::sig` module includes the primitive operations for
implementing signature verification.
Dustin