jenkins: Migrate to iSCSI persistent volume

Managing the Jenkins volume with Longhorn has become increasingly
problematic.  Because of its large size, whenever Longhorn needs to
rebuild/replicate it (which happens often for no apparent reason), it
can take several hours.  While the synchronization is happening, the
entire cluster suffers from degraded performance.

Instead of using Longhorn, I've decided to try storing the data directly
on the Synology NAS and expose it to Kubernetes via iSCSI.  The Synology
offers many of the same features as Longhorn, including
snapshots/rollbacks and backups.  Using the NAS allows the volume to be
available to any Kubernetes node, without keeping multiple copies of
the data.

In order to expose the iSCSI service on the NAS to the Kubernetes nodes,
I had to make the storage VLAN routable.  I kept it as IPv6-only,
though, as an extra precaution against unauthorized access.  The
firewall only allows nodes on the Kubernetes network to access the NAS
via iSCSI.

I originally tried proxying the iSCSI connection via the VM hosts,
however, this failed because of how iSCSI target discovery works.  The
provided "target host" is really only used to identify available LUNs;
follow-up communication is done with the IP address returned by the
discovery process.  Since the NAS would return its IP address, which
differed from the proxy address, the connection would fail.  Thus, I
resorted to reconfiguring the storage network and connecting directly
to the NAS.

To migrate the contents of the volume, I temporarily created a PVC with
a different name and bound it to the iSCSI PersistentVolume.  Using a
pod with both the original PVC and the new PVC mounted, I used `rsync`
to copy the data.  Once the copy completed, I deleted the Pod and both
PVCs, then created a new PVC with the original name (i.e. `jenkins`),
bound to the iSCSI PV.  While doing this, Longhorn, for some reason,
kept re-creating the PVC whenever I would delete it, no matter how I
requested the deletion.  Deleting the PV, the PVC, or the Volume, using
either the Kubernetes API or the Longhorn UI, they would all get
recreated almost immediately.  Fortunately, there was actually enough of
a delay after deleting it before Longhorn would recreate it that I was
able to create the new PVC manually.  Once I did that, Longhorn seemed
to give up.

jenkins/.gitignore

@@ -0,0 +1 @@
+iscsi-chap.yaml

jenkins/README.md

@@ -29,3 +29,11 @@ Clouds*:
 
 
 [0]: https://plugins.jenkins.io/kubernetes/
+
+
+## iSCSI Persistent Volume
+
+Because of the large size of the Jenkins volume, it does not work well managed
+by Longhorn.  Instead, we use a pre-provisioned iSCSI volume on the Synology
+NAS.  This improves performance and avoids keeping multiple replicas of the
+Jenkins data, while still benefiting from snapshots, etc.

jenkins/iscsi-migrate.yaml

@@ -0,0 +1,51 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: jenkins2
+  namespace: jenkins
+spec:
+  accessModes:
+  - ReadWriteOnce
+  storageClassName: ''
+  volumeName: jenkins
+  resources:
+    requests:
+      storage: 40G
+
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: migrate
+  namespace: jenkins
+spec:
+  nodeSelector:
+    kubernetes.io/arch: amd64
+  containers:
+  - image: git.pyrocufflink.net/containerimages/dch-debug
+    name: migrate
+    command:
+    - rsync
+    args:
+    - -aiHAXS
+    - /mnt/jenkins/
+    - /mnt/jenkins2/
+    - --exclude=lost+found
+    securityContext:
+      runAsUser: 0
+      runAsGroup: 0
+      seLinuxOptions:
+        level: s0:c525,c600
+    volumeMounts:
+    - mountPath: /mnt/jenkins
+      name: jenkins
+    - mountPath: /mnt/jenkins2
+      name: jenkins2
+  restartPolicy: Never
+  volumes:
+  - name: jenkins
+    persistentVolumeClaim:
+      claimName: jenkins
+  - name: jenkins2
+    persistentVolumeClaim:
+      claimName: jenkins2

jenkins/iscsi.yaml

@@ -0,0 +1,50 @@
+apiVersion: bitnami.com/v1alpha1
+kind: SealedSecret
+metadata:
+  name: iscsi-chap
+  namespace: jenkins
+  annotations:
+    app.kubernetes.io/name: iscsi-chap
+    app.kubernetes.io/component: jenkins
+    app.kubernetes.io/part-of: jenkins
+spec:
+  encryptedData:
+    node.session.auth.password: AgAR1jsfJ0/jzQBwBXhbes8xI30qGjCtI20Zny1cf4vh39xdS28PGok2B9VEMFaZwit8PKCVecPo+Xfc/KBQCx57kfkRjfOEbSr32sYsT/rdtldYQwLYuDZ9hT9tto4cXFcMSKWQwPMdCuqF0vn4M2mhCcs0KyMNpemGqkPux0maAa6wgKNNGgNitg/EymDVhZBYflQxA8E+JXVrdvlj6wmRr5WW/3Xx/yWUlGQfiZeihORm/Ab+CL2p99LpGZVLitiL3tsMly19/ibt0OU6pkaKnL9Rb7EBxcdpYdRRVDbKyuyGRPyX1vsTM4u5IpX2HmXW4jRJQxpwnzQ2dcthQyKIh7IkezeiFOeHh+AOfo3lmF2nHOMFZmb+848G02+3qYDnGBzMTaZ/gWjjtR9ronlCSCH1drUQ7YIOWsW3anqKJwZs+oqbZddA9hW8ya6y4cRxcKqloFQteXI4EIBuJii2BRCsvg6zHExARZhHZMf6B3SEW9UjRDDJHVOiFg8tJP2UAsLm6yOsYUDE1Ld8JeLz7NvyPA/M4UtuGyI8nNDlv83nPZOyYq/h9gRHp4TG7Qo4YZDFRMdV1soz51WI1wUOzXRZD8Tia5CleDxN9fiyLpVnC8Z38AhIo4yVByjjTIV471a67ta2U0zoHQ/gqxrq8G+bkrP55ygXCiDybVOJrcS1jPO5UUtRa5H5GBhbQFQ5Q5X9eRQ+Qmqm12ScRYD4
+    node.session.auth.password_in: AgA+2CglEAZH7NQvhzZWWFs+IvZsVNCy58BX/5PLTIgyaFLlXUW7tyoA49CcnTAtYqxcTLSvqcRpstjGtl5Eq+y08mzC0VSAytiRWMJ9fZwq+h4eQEfabPFtMNVZOVMm+0c8NADWD8PLkIWb6yp6QbGv4uN+Abo+uWXgTPHCw+8TOMcYxs5RcSPjkg3jvCJuCZi8IuTmUMzCiCrpZMSTNZaGh4jD0tJjrBWFwvRnkFlgy2skMNY0LoPZ4ZmYp+KGE1IK/Soom22xOwG7NdKDMHYD4hZrflAqLBKcEb0AiM12j3v4UoQSUfZ4+KcTZvtgSf127HaivS34w8payZANh9izzzxZlwA2wc3GacCFzQrpDIsRI9QgDrxGDNFSwBLzFjmWMD/eWLsY6xgZnS5Q5wHiCW10t+13KHvhl59ovf6UEjtCwH14dfMn/qMu+Sd8tqrZGV5dSHLyFD5PufGkuMw3G6YTsTRU5APdShbnhs8StLccEI6drYjotn7g7xDyzlHEuswc8kD//W8PDyXTBenSgLHeu61Ud1O843KjVcPxdlIFaRGX1EsQUj+zXm8v1A4Ixfm07JRn03FfurJy/NKhKsqvrrafXGNXTh3CLlHObdk3Uqj9IxbDUqzcuZ8+sL9Ia/NW9gN7CRUa1ARfMbrZ/uzNSEulo8vqS3M3DKk+xNm+ivqt4ZEVDOYrHwBGM/HOxDr5TOvck/SSVax8dv3y
+    node.session.auth.username: AgAFfDEVU9BJF085N3V64AP6ZU3ImY7gIsuqqfEbrTOekz9jyvQkrsA1mTNUmfnvD6oJJGv2XYGjxzPsiw//YZSUwmXLgYMBzHYWRl4yFvDmNUo3W3au9rUUha1WxBQ/s7V8Qz0ucoaYhpaMEXZ39OguLzpwYpiCAiBLn10JlGAWFyPfRRYl5Ybt/WpdHV/b0pog6HKO64xWhIu+jxa4KwszfYbYQ0HNnHHsGZpGukMwFLs/RHb+G/ob/5jjhW7wqf1qfuotNJ6scgIHTv+N8Zd057Yfg+syNv4GBECTjH7xXAKKDUJ31pORRI7mDyuq7s24IqJHYwZEMxN2+Pa9l24f73hIa1ZJLS2FDTK4C6mWRvTFwlYm4KSYp3OJgHW/+78wkAIjKOokn5BUKaArEZqUYiftjFu0js1WVZ1AF2ZasH0PdCqm8GHC9nAG39+ZKNgoVF2SC1In9a7/RtxV3VEG5D9o5E7FoZoLF6RXWfn3k1xeFYJ+RZwcLKw9hIe8lrEOZzA1ikGLijUShKZrT1wH4Hh+lMMDqld0kga406BtEJaBCIoNGdvtDLs1IxO8znE+EDIjd8bQwbEBzzigeIrL5B9+F7jeDqD+4wxCs76cJzvrj2AKRR3RtCs8t0nDrfOmE2X7wOGZXjypBmAYGeNww0Y8Y5CeqIfdHW/x1Pe8zR9PMzoE3OFHstepHj8451SpVAKsQj+9
+    node.session.auth.username_in: AgBREdELfPLFznIgcaB8kaOTef3MSDz7FsuHBZ+PJNR8mOmbFeo6K+D6Kx+tPHXd9sLect9q5gPYHqRtJLHcvEL/YU566nUodn+DDudlLrjJB5x7kf8dXkYXzUpEznq/nF6OLNZjEntjb5FqtPRJQY9JV+zTF9hSGJiC++7rDHrBTeBWXu/I+czoOu6Zx1N5r6f9DMTKkr9YjRBawPVfo4ySR5Zq2nZAwI0/HwAaBaOgVdpxpLSlZ29W761Gd43kxz11ngctRWR8BjPFhVI8JOv7zaTex65gpDB7YPAd1tZobVxtKUijrRa45N6j6VqFYq3+t1vVFjFPoGM/DUKqd9XClSUkdENlGZB+71UVoR6qLiNRQejA6LNN/ZycGQlTDRYr1wljTYeUu//x/ZouPKWSD0jAh91Z9qL0wh+L5gghOSS2vkqU+Vsbtl3E9PZ+yHK3jE9+fuYBmrHdUWovNpQMkk+9Rdhhv6oET2zzbl9BwQgnuF1LCO+GLmEhYxfUlwR3Ki9i1+PuvBx2NJyr2ukRf1uiH5WoQDs2eJEEmM1LtzYf+mzWXR752zpM8GaMAA6CcWbIw5XpuCaAEfOalgHnwIPJE2pK2AZv4n8hYGqWhYqQizVHHLm1dyllzYiB2kaJwUOlTlfWzQ9mXrME7KCbMhuMVvGTG6bn87ER1C9w+bCmk93J1xG6EFwhG6fU3n8zpNZ1AzNEAA==
+  template:
+    metadata:
+      name: iscsi-chap
+      namespace: jenkins
+      annotations:
+        app.kubernetes.io/name: iscsi-chap
+        app.kubernetes.io/component: jenkins
+        app.kubernetes.io/part-of: jenkins
+
+---
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: jenkins
+  namespace: jenkins
+spec:
+  accessModes:
+  - ReadWriteOnce
+  storageClassName: ''
+  capacity:
+    storage: 40G
+  iscsi:
+    # Has to be an IP address, even though the documentation says it can be a
+    # hostname.  Otherwise, error: "Could not get SCSI host number for
+    # portal"
+    targetPortal: '[fd68:c2d2:500e:3ea3:8d42:e33e:264b:7c30]:3260'
+    iqn: iqn.2000-01.com.synology:storage0.jenkins.8181625090
+    lun: 1
+    # Synology does not require CHAP for discovery/send_targets
+    chapAuthDiscovery: false
+    chapAuthSession: true
+    fsType: ext4
+    secretRef:
+      name: iscsi-chap
+

jenkins/kustomization.yaml

@@ -9,6 +9,7 @@ resources:
 - jenkins.yaml
 - argocd-sync-hook.yaml
 - secrets.yaml
+- iscsi.yaml
 
 configMapGenerator:
 - name: ssh-known-hosts
@@ -17,3 +18,14 @@ configMapGenerator:
   - ssh_known_hosts
   options:
     disableNameSuffixHash: true
+
+patches:
+- patch: |
+    apiVersion: v1
+    kind: PersistentVolumeClaim
+    metadata:
+      name: jenkins
+      namespace: jenkins
+    spec:
+      volumeName: jenkins
+      storageClassName: ''