Upstream changed the naming convention for Fedora AMIs. It also seems
they've stopped publishing "release" artifacts; all the AMIs are now
date-stamped. We should probably consider running `terraform apply`
periodically to keep up-to-date.
dustin/dynk8s-provisioner/pipeline/head This commit looks goodDetails
If a Jenkins job runs for a while, Kubernetes may schedule other Pods
on it eventually. If a long-running Pod gets assigned to the ephemeral
node, the Cluster Autoscaler won't be able to scale down the ASG. To
prevent this, we apply a taint to the node so normal Pods will not get
assigned to it. We have to apply the corresponding toleration to Pods
for Jenkins jobs.
Fedora AMIs have the default locale set to en_US.UTF-8, which sorts
`100-crio-bridge.conflist` before `10-calico.conflist`. As a result,
Pods end up with incorrect network configuration, and cannot be reached
from other Pods on the container network. Since we do not need the
default configuration, the easiest way to resolve this is to just delete
it.
dustin/dynk8s-provisioner/pipeline/head This commit looks goodDetails
Jenkins jobs that build container images in user namespaces need access
to `/dev/fuse`, which is provided by the [fuse-device-plugin][0]. This
plugin runs as a DaemonSet, which updates the status of the node it's
running on when it starts to indicate that the FUSE device is available.
When scaling up from zero nodes, Cluster Autoscaler has no way to know
that this will occur, and therefore cannot determine that scaling up the
ASG will create a node with the required resources. Thus, the ASG needs
a tag to inform CA that the nodes it creates will indeed have the
resources and scaling it up will allow the pod to be scheduled.
Although this feature of CA was added in 1.14, it apparently got broken
at some point and no longer works in 1.22. It works again in 1.26,
though.
[0]: https://github.com/kuberenetes-learning-group/fuse-device-plugin/tree/master
The *cri-o* package has moved from its own module into the base Fedora
repository, as Fedora is [eliminating modules][0]. The last modular
version was 1.25, which is too old to run pods with user namespaces.
Version 1.26 is available in the base repository, which does support
user namespaces.
[0]: https://fedoraproject.org/wiki/Changes/RetireModularity
Lately, cloud nodes seem to be failing to come up more frequently. I
traced this down to the fact that `/etc/resolv.conf` in the `kube-proxy`
container contains both the AWS-provided DNS server and the on-premises
server set by Wireguard. This evidently "works" correctly sometimes,
but not always. When it doesn't, the `kube-proxy` cannot resolve the
Kubernetes API server address, and thus cannot create the necessary
netfilter rules to forward traffic correctly. This causes pods to be
unable to communicate.
I am not entirely sure what the "correct" solution to this problem would
be, since there are various issues in play here. Fortunately, cloud
nodes are only ever around for a short time, and never need to be
rebooted. As such, we can use a "quick fix" and simply remove the
AWS-provided DNS configuration.
The Cluser Autoscaler uses EC2 Auto-Scaling Groups to configure the
instances it launches when it determines additional worker nodes are
necessary. Auto-Scaling Groups have an associated Launch Template,
which describes the properties of the instances, such as AMI ID,
instance type, security groups, etc.
When instances are first launched, they need to be configured to join
the on-premises Kubernetes cluster. This is handled by *cloud-init*
using the configuration in the instance user data. The configuration
supplied here specifies the Fedora packages that need to be installed on
a Kubernetes worker node, plus some additional configuration required by
`kubeadm`, `kubelet`, and/or `cri-o`. It also includes a script that
fetches the WireGuard client configuration and connects to the VPN,
finalizes the setup process, and joins the cluster.
The `terraform` directory contains the resource descriptions for all AWS
services that need to be configured in order for the dynamic K8s
provisioner to work. Specifically, it defines the EventBridge rule and
SNS topic/subscriptions that instruct AWS to send EC2 instance state
change notifications to the *dynk8s-provisioner*'s HTTP interface.
Dustin