The *jenkins-repohost* Secret contains an SSH private key Jenkins jobs
can use to publish RPM packages to the Yum repo host on
*files.pyrocufflink.blue*.
The *rpm-gpg-key* and *rpm-gpg-key-passphrase* Secrets contain the GnuPG
private key and its encryption passphrase, respectively, that can be
used to sign RPM packages. This key is trusted by managed nodes on the
Pyrocufflink network.
The [Kubernetes Credentials Provider][0] plugin for Jenkins allows
Jenkins to expose Kubernetes Secret resources as Jenkins Credentials.
Jobs can use them like normal Jenkins credentials, e.g. using
`withCredentials`, `sshagent`, etc. The only drawback is that every
credential exposed this way is available to every job, at least until
[PR #40][1] is merged. Fortunately, jobs managed by this Jenkins
instance are all trusted; no anonymous pull requests are possible, so
the risk is mitigated.
[0]: https://jenkinsci.github.io/kubernetes-credentials-provider-plugin/
[1]: https://github.com/jenkinsci/kubernetes-credentials-provider-plugin/pull/40
Setting the `imagePullSecrets` property on the default service account
for the *jenkins-jobs* namespace allows jobs to run from private
container images automatically, without additional configuration in the
pipeline definitions.
The Raspberry Pi usually has the most free RAM of all the Kubernetes
nodes, so pods tend to get assigned there even when it would not be
appropriate. Jenkins, for example definitely does not need to run
there, so let's force it to run on the bigger nodes.
Argo CD will delete and re-create this Job each time it synchronizes the
*jenkins* application. The job creates a snapshot of the Jenkins volume
using an HTTP request to the Longhorn UI.
When cloning/fetching a Git repository in a Jenkins pipeline, the Git
Client plugin uses the configured *Host Key Verification Strategy* to
verify the SSH host key of the remote Git server. Unfortunately, there
does not seem to be any way to use the configured strategy from the
`git` command line in a Pipeline job, so e.g. `git push` does not
respect it. This causes jobs to fail to push changes to the remote if
the container they're using does not already have the SSH host key for
the remote in its known hosts database.
This commit adds a ConfigMap to the *jenkins-jobs* namespace that can be
mounted in containers to populate the SSH host key database.
I don't want Jenkins updating itself whenever the pod restarts, so I'm
going to pin it to a specific version. This way, I can be sure to take
a snapshot of the data volume before upgrading.
Setting a static SELinux level for the container allows CRI-O to skip
relabeling all the files in the persistent volume each time the
container starts. For this to work, the pod needs a special annotation,
and CRI-O itself has to be configured to respect it:
```toml
[crio.runtime.runtimes.runc]
allowed_annotations = ["io.kubernetes.cri-o.TrySkipVolumeSELinuxLabel"]
```
This *dramatically* improves the start time of the Jenkins container.
Instead of taking 5+ minutes, it now starts instantly.
https://github.com/cri-o/cri-o/issues/6185#issuecomment-1334719982
Running Jenkins in Kubernetes is relatively straightforward. The
Kubernetes plugin automatically discovers all the connection and
authentication configuration, so a `kubeconfig` file is no longer
necessary. I did set the *Jenkins tunnel* option, though, so that
agents will connect directly to the Jenkins JNLP port instead of going
through the ingress controller.
Jobs now run in pods in the *jenkins-job* namespace instead of the
*jenkins* namespace. The latter is now where the Jenkins controller
runs, and the controller should not have permission to modify its own
resources.
Jenkins doesn't really need full control of all resources in its
namespace. Rather, it only needs to be able to manage Pod and
PersistentVolumeClaim resources.
Dustin