Dustin C. Hatch
f7f408ca8c
Longhorn does not work well for very large volumes. It takes ages to synchronize/rebuild them when migrating between nodes, which happens all too frequently. This consumes a lot of resources, which impacts the operation of the rest of the cluster, and can cause a cascading failure in some circumstances. Now that the cluster is set up to be able to mount storage directly from the Synology, it makes sense to move the Victoria Metrics data there as well. Similar to how I did this with Jenkins, I created PersistentVolume resources that map to iSCSI volumes, and patched the PersistentVolumeClaims (or rather the template for them defined by the StatefulSet) to use these. Each `vmstorage` pod then gets an iSCSI LUN, bypassing both Longhorn and QEMU to write directly to the NAS. The migration process was relatively straightforwrad. I started by scaling down the `vminsert` Deployment so the `vmagent` pods would queue the metrics they had collected while the storage layer was down. Next, I created a [native][0] export of all the time series in the database. Then, I deleted the `vmstorage` StatefulSet and its associated PVCs. Finally, I applied the updated configuration, including the new PVs and patched PVCs, and brought the `vminsert` pods back online. Once everything was up and running, I re-imported the exported data. [0]: https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-export-data-in-native-format
Dustin's Kubernetes Cluster
This repository contains resources for deploying and managing my on-premises Kubernetes cluster
Cluster Setup
The cluster primarily consists of libvirt/QEMU+KVM virtual machines. The Control Plane nodes are VMs, as are the x86_64 worker nodes. Eventually, I would like to add Raspberry Pi or Pine64 machines as aarch64 nodes.
All machines run Fedora, using only Fedora builds of the Kubernetes components
(kubeadm, kubectl, and kubeadm).
See Cluster Setup for details.
Jenkins Agents
One of the main use cases for the Kubernetes cluster is to provide dynamic agents for Jenkins. Using the Kubernetes Plugin, Jenkins will automatically launch worker nodes as Kubernetes pods.
See Jenkins Kubernetes Integration for details.
Persistent Storage
Persistent storage for pods is provided by Longhorn. Longhorn runs within the cluster and provisions storage on worker nodes to make available to pods over iSCSI.
See Persistent Storage Using Longorn for details.