Self hosted Github Actions runners in Kubernetes
Published
Since August last year, Github Actions has evolved into a complete solution to build pipelines for CI/CD, as well as a wide variety of other automation tasks. It's not always been the most reliable CI/CD solution available, but the service has improved a lot over time and it's a reasonably good option now. Writing YAML workflows for Github Actions is pretty easy, and you get 2000 minutes of usage of Github's hosted runners included with the free Github plan, which is plenty for small teams. After that, pricing is affordable and competitive compared to that of other solutions in the market.
Besides the ease of use and the overall good design, one thing I particularly like is that you can also self host Github Actions runners in your own infrastructure for absolutely no cost. It's totally free, and you can use as beefy servers for this as you like, which can translate in better performance when executing workflows compared to Github's own runners which only have 2 cores and 7 GB of memory. Self hosted runners allow for greater control, making workflows more flexible. Also, if your workflows build Docker images, you need some workarounds to benefit from layer caching with Github's own runners since each time a workflow runs a brand new environment is spun up, so images built with previous runs are not cached out of the box. By using a self hosted runner instead, you can choose to use the same Docker instance across workflow runs, so new workflow runs can leverage layers cached in previous builds, which can dramatically speed up the execution.
I have been using a self hosted runner for Github Actions for a little while now, but while until recently I was using a VPS dedicated to the job, I am now leveraging some spare capacity in my Kubernetes cluster for this purpose to build, test and deploy DynaBlogger; this post is about how to set up self hosted runners for Github Actions in Kubernetes for workflows that require building Docker images. I'll assume you are already familiar with how Github Actions work.
There are various ways to build Docker images inside a Kubernetes cluster, but perhaps the easiest is with the "Docker in Docker" approach available with the official Docker image - it's basically the ability to run a Docker daemon inside a container and you can run the usual build commands as if you ran these commands directly on the actual host. We'll need a deployment of Docker in Docker ("dind") as well as a deployment of an image containing all the required dependencies to run self hosted runners for Github Actions which will leverage the dind deployment. This has a couple of important benefits:
- we can use a persistent volume for dind, so to benefit from layer caching across multiple workflow runs
- we can easily scale runners by just scaling the deployment, allowing for multiple workflows to be executed in parallel.
Setting up the Docker in Docker deployment
For dind we'll use a regular Kubernetes deployment and the dind version of the official Docker image. First things first, you'll need to create a namespace - I name this namespace as docker-in-docker:
kubectl create ns docker-in-dockerNext, we need to create a persistent volume claim (pvc) in this namespace that dind will use to persist the Docker layers even if the pod is rescheduled for any reason:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: dind
namespace: docker-in-docker
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
I am creating a volume of 50GB here since that should be enough for me (I can just exec into the container and prune images if needed every now and then), but you are free to create a larger or smaller volume.
Like I said for dind we'll need a regular deployment, so create deployment-dind.yml with the following:
apiVersion: apps/v1
kind: Deployment
metadata:
name: dind
namespace: docker-in-docker
spec:
replicas: 1
selector:
matchLabels:
workload: deployment-docker-in-docker-dind
template:
metadata:
labels:
workload: deployment-docker-in-docker-dind
spec:
containers:
- command:
- dockerd
- --host=unix:///var/run/docker.sock
- --host=tcp://0.0.0.0:2376
env:
- name: DOCKER_TLS_CERTDIR
image: docker:19.03.12-dind
imagePullPolicy: IfNotPresent
name: dind
resources: {}
securityContext:
privileged: true
readOnlyRootFilesystem: false
stdin: true
tty: true
volumeMounts:
- mountPath: /var/lib/docker
name: dind-storage
volumes:
- name: dind-storage
persistentVolumeClaim:
claimName: dindA few things to note here:
- we are setting the command explicitly because by default dind will run with TLS for the connections between the daemon and the clients. You can either leave the command unset to use TLS or override it to ignore TLS to make things a bit easier, it's up to you. I choose to disable TLS for this. Besides specifying a command without the default TLS arguments, we also need to set the DOCKER_TLS_CERTDIR environment variable to an empty string;
- the image we are using is the latest docker image with the dind tag, so to be able to make the Docker daemon available to remote clients;
- I am leaving the resources unset in this cluster, but since CPU-intensive workflows might use a lot of CPU, you may want to set some limits here;
- finally, we are making sure that the pod can use the persistent volume claim we created earlier.
To deploy, run:
kubectl apply -f deployment-dind.ymlSetting up the Github Actions runners' deployment
The next thing we need to do, is set up the deployment for the actual runners. I found a few different ways to do this, but the easiest for me was using this repo since it already includes a useful Dockerfile as well as a sample deployment manifest for kubernetes. So go ahead and clone the repository, then open the Dockerfile in your editor.
The Dockerfile installs the dependencies required by Github Actions to connect the runner to the Github Actions service, so that the runner can listen for jobs and report progress to Github. We can build a custom image using this Dockerfile and that's what we'll do in order to make using the dind instance possible. In particular we need to add the following to the apt install command:
&& curl https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz --output docker-19.03.9.tgz \
&& tar xvfz docker-19.03.9.tgz \
&& cp docker/* /usr/bin/The final Dockerfile will look something like the following:
FROM debian:buster-slim
ARG GITHUB_RUNNER_VERSION="2.273.4"
ENV RUNNER_NAME "runner"
ENV GITHUB_PAT ""
ENV GITHUB_OWNER ""
ENV GITHUB_REPOSITORY ""
ENV RUNNER_WORKDIR "_work"
ENV RUNNER_LABELS ""
ENV DOCKER_HOST="tcp://docker-in-docker.dind:2376"
RUN apt-get update \
&& apt-get install -y \
curl \
sudo \
git \
jq \
iputils-ping \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& useradd -m github \
&& usermod -aG sudo github \
&& echo "%sudo ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers \\
&& curl https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz --output docker-19.03.9.tgz \
&& tar xvfz docker-19.03.9.tgz \
&& cp docker/* /usr/bin/
USER github
WORKDIR /home/github
RUN curl -Ls https://github.com/actions/runner/releases/download/v${GITHUB_RUNNER_VERSION}/actions-runner-linux-x64-${GITHUB_RUNNER_VERSION}.tar.gz | tar xz \
&& sudo ./bin/installdependencies.sh
COPY --chown=github:github entrypoint.sh ./entrypoint.sh
RUN sudo chmod u+x ./entrypoint.sh
ENTRYPOINT ["/home/github/entrypoint.sh"]
One important thing to note here is that whenever there is a new release of the runner's software, we'll need to rebuild our image and update the deployment. The reason is that the runner itself will listen not only to jobs for workflows to run, but also to notifications whenever a new version of its software is available, which triggers a download of the new software followed by a restart of the runner. Of course, in Kubernetes this will cause the pod to restart, and then the runner will download the new software and restart again, causing a crash loop. To prevent that, I recommend you watch the runner's code repo so you get notified when a new version is available, and can rebuild the image.
Next, go ahead and build the image, then push it to your registry of choice. If you don't need to change anything in the Dockerfile, you can just use the image I prebuilt and published on Docker Hub, vitobotta/github-actions-runner:0.0.4.
You don't need to change anything in the entrypoint.sh script, which simply installs the runner's software. However you need to make a few small changes to the deployment manifest deployment.yml so that it looks like the following:
apiVersion: apps/v1
kind: Deployment
metadata:
name: github-runner
namespace: github-actions
labels:
app: github-runner
spec:
replicas: 1
selector:
matchLabels:
app: github-runner
template:
metadata:
labels:
app: github-runner
spec:
containers:
- name: github-runner
image: vitobotta/github-actions-runner:0.0.4
env:
- name: DOCKER_HOST
value: tcp://dind.docker-in-docker:2376
- name: GITHUB_OWNER
value: <your github username>
- name: GITHUB_REPOSITORY
value: <your repository>
- name: GITHUB_PAT
valueFrom:
secretKeyRef:
name: github-actions-token
key: pat
As you can see we set the DOCKER_HOST to our dind pod. You can increase the number of replicas if you want to be able to run multiple workflows in parallel.
We need to do a few things before we can deploy the runner image. First, we need to create a namespace:
kubectl create ns github-actionsThen, you need to create a personal access token in Github (Settings > Developer Settings > Personal Access Tokens) - remember to add full repo permissions to this token. Next, create a secret in Kubernetes with the token in the github-actions namespace:
apiVersion: v1
stringData:
pat: <your token>
kind: Secret
metadata:
name: github-actions-token
namespace: github-actions
type: OpaqueWe can now deploy:
kubectl apply -f deployment.ymlWait for the pod to restart, then head to Github, and in the Settings > Actions page of your repository you should see a newly created, idle runner.
The runner is basically ready, and you can update your workflows so that they can use it instead of Github's own runners:
...
jobs:
build:
runs-on: [self-hosted]
...The next time you push code to your repository, the new self hosted runner should pick up the job from Github automatically and execute it. You can check this from looking at the logs of the runner's pod.
Conclusions
As you can see, setting up self hosted runners for Github Actions in Kubernetes is pretty straightforward, requiring only a couple of simple deployments. I recently set up pipelines in Semaphore CI following a couple of outages with Github Actions, but I am back to using Github Actions because my builds are faster with my own runners (my nodes have 8 cores vs the 2 cores of Github's runners, and a lot more memory), and of course you cannot beat free as far as price is concerned. I'm sure that the reliability of the service will further improve over time, so for now I will stick with it. One thing I don't like though is that you need to set up separate runners for each repository - you cannot configure runners for the entire Github account or organisation.
Let me know in the comments if you have any questions or run into any issues when setting things up.
I am passionate about WebDev, DevOps and CyberSecurity. I am based in Espoo, Finland, where I work with the backend team at 