Instead of hard-coding the AMI ID of the Fedora build we want, we can
use the `aws_ami` data source to search for it. The Fedora release team
has a consistent naming scheme for AMIs, so finding the correct one is
straightforward.
Lately, cloud nodes seem to be failing to come up more frequently. I
traced this down to the fact that `/etc/resolv.conf` in the `kube-proxy`
container contains both the AWS-provided DNS server and the on-premises
server set by Wireguard. This evidently "works" correctly sometimes,
but not always. When it doesn't, the `kube-proxy` cannot resolve the
Kubernetes API server address, and thus cannot create the necessary
netfilter rules to forward traffic correctly. This causes pods to be
unable to communicate.
I am not entirely sure what the "correct" solution to this problem would
be, since there are various issues in play here. Fortunately, cloud
nodes are only ever around for a short time, and never need to be
rebooted. As such, we can use a "quick fix" and simply remove the
AWS-provided DNS configuration.
The default configuration for the *kubelet.service* unit does not
specify the path to the `config.yml` generated by `kubeadm`. Thus, any
settings defined in the `kublet-config` ConfigMap do not take effect.
To resolve this, we have to explicitly set the path in the `config`
property of the `kubeletExtraArgs` object in the join configuration.
The Cluser Autoscaler uses EC2 Auto-Scaling Groups to configure the
instances it launches when it determines additional worker nodes are
necessary. Auto-Scaling Groups have an associated Launch Template,
which describes the properties of the instances, such as AMI ID,
instance type, security groups, etc.
When instances are first launched, they need to be configured to join
the on-premises Kubernetes cluster. This is handled by *cloud-init*
using the configuration in the instance user data. The configuration
supplied here specifies the Fedora packages that need to be installed on
a Kubernetes worker node, plus some additional configuration required by
`kubeadm`, `kubelet`, and/or `cri-o`. It also includes a script that
fetches the WireGuard client configuration and connects to the VPN,
finalizes the setup process, and joins the cluster.
The `terraform` directory contains the resource descriptions for all AWS
services that need to be configured in order for the dynamic K8s
provisioner to work. Specifically, it defines the EventBridge rule and
SNS topic/subscriptions that instruct AWS to send EC2 instance state
change notifications to the *dynk8s-provisioner*'s HTTP interface.
Dustin