has_many :codes

Kubernetes on Hetzner Cloud the easiest way


A big percentage of the visits to this blog are for articles I have written over time about Kubernetes, and in particular on setting up Kubernetes in Hetzner (referral link, we both receive credits) since that's the provider I have used the most for years for personal projects (now they even have 2 locations in the USA, so they are becoming more popular every day thanks to that; I highly recommend this provider because it combines an excellent service with amazing prices) . In these articles I have mainly described how to set things up using Kubernetes distributions created by Rancher, that is RKE and k3s.

k3s has become my favorite way to deploy Kubernetes because it's simpler (just a single binary!) and it's very fast to install, compared to other distros based on vanilla Kubernetes. There are tools like k3sup by Alex Ellis that make it easy to quickly deploy k3s on existing nodes, but you still need to create those nodes in advance, since the tool doesn't provision infrastructure automatically for you. There are various other ways to deploy Kubernetes to Hetzner Cloud servers, for example based on Terraform, KOPS and more, which can also provision the infrastructure, but I was looking for a tool that

- creates a sound infrastructure configuration in Hetzner Cloud out of the box, automatically and very quickly
- quickly deploys Kubernetes to all the nodes
- sets up whatever is required to be able to provision load balancers and persistent volumes out of the box
- makes upgrading to a new version of Kubernetes easy and quick
- implements the Cluster Autoscaler so that I don't have to worry too much about sizing my clusters
- is as easy to use as possible, with as little configuration as possible

I couldn't find a single tool that combines all the above requirements, so I set to build one, which I called hetzner-k3s. As you may guess, my tool, which is a CLI written in Crystal, deploys k3s on Hetzner Cloud infrastructure and satisfies all the requirements I had beautifully.

Creating a production grade cluster with a highly available control plane (3 masters), plus 10 worker nodes in multiple locations also for higher availability, takes around 3-4 minutes, which I think is awesome. You may be more interested in how fast upgrades are though, and whether upgrades cause downtime and if yes, for how long. The beauty of k3s is that it being a single binary, it's much quicker to upgrade a cluster compared to standard Kubernetes. hetzner-k3s installs Rancher's System Upgrade Controller which makes performing rolling upgrades to a new version of k3s a breeze. Upgrades are really fast and there is no downtime concerning your workloads during upgrades. Of course, if you upgrade a cluster with a single master the control plane will be briefly unavailable during the upgrade, but it literally takes seconds so it's not a big deal.

More details about hetzner-k3s:

- it creates all the infrastructure required, and quickly. This means: the cloud servers for masters and workers, a private network (to make sure all the traffic between nodes goes through a private network), a firewall to restrict access to the cluster, and - for HA clusters with multiple masters - a load balancer for the Kubernetes API
- it deploys k3s, of course. It sets up the first master, followed by the other masters if it's an HA cluster, concurrently (to speed up the process) and then all the worker nodes, also concurrently
- for improved security, it allows you to specify which network or networks can access the nodes via SSH, or the Kubernetes API (there are separate settings for each of these)
- it allows you to enable or disable scheduling of regular workloads on masters (for example, in a production cluster you may want your workloads to run only on worker nodes)
- it allows you to configure one or more worker node pools, even in different locations for higher availability
- it allows you to enable autoscaling on worker node pools, so you don't have to worry about resizing the cluster capacity manually depending on your workloads' needs in terms of CPU and memory
- it allows you to enable encryption as an added layer of security, even if the traffic between nodes goes through a private network
- it allows you to specify additional packages that should be installed on the nodes (for example to add support for some specific features your workloads require)
- it allows you to specify additional commands that should be run on the nodes after they are created (to upgrade the system or perform some tasks that are required by workloads)
- it allows you to specify taints and labels for the nodes, for example to deploy some workloads on specific nodes
- it allows you to specify extra arguments for the various Kubernetes components (API server, scheduler, controller manager, cloud controller manager, kubelet, kube proxy)
- it installs the Hetzner Cloud Controller Manager, which among other things makes it possible to provision load balancers for your workloads
- it installs the Hetzner CSI driver, which makes it possible to provision persistent volumes using Hetzner Cloud's block storage feature (these volumes are based on Ceph so they are replicated and highly available, with very good performance)
- it installs the aforementioned upgrade controller, to make upgrades easy and quick
- it installs the Cluster Autoscaler, so you can enable autoscaling on one or more node pools. I particularly love this feature, because makes clusters created with this tool behave like managed Kubernetes clusters

Creating a cluster

Creating a cluster with hetzner-k3s could not be simpler, and is certainly simpler than with other available tools in my opinion. First, assuming you already have an Hetzner Cloud account, go to Security in the Hetzner Cloud control panel, and select the API Tokens tab. Then create a token with both read and write permissions and take note of it because you will only see it once.

Next, create a simple config file like in the example below, named e.g. cluster.yaml:

hetzner_token: <your token>
cluster_name: test
kubeconfig_path: "./kubeconfig"
k3s_version: v1.25.5+k3s1
public_ssh_key_path: "~/.ssh/id_ed25519.pub"
private_ssh_key_path: "~/.ssh/id_ed25519"
use_ssh_agent: false
schedule_workloads_on_masters: false
  instance_type: cx21
  location: nbg1
  instance_count: 3
- name: fsn-cpx21
  instance_type: cpx21
  location: fsn1
  instance_count: 1
- name: autoscaled-hel-cpx31
  instance_type: cpx31
  location: hel1
    enabled: true
    min_instances: 0
    max_instances: 5

The configuration above is what is required - see the README in the repository for all the available configuration options. A few notes on the configuration above:

- in this example, kubeconfig_path is set to a file in the current directory, but you can specify any valid path like ~/.kube/config if you want this cluster to be the default cluster
- use_ssh_agent must be set to true if you are using an SSH key with a passphrase, and you need to use an SSH agent (how to do it depends on your OS); the public SSH key will be added to each node so that you can SSH into them later, if ever needed
- ssh_allowed_networks and api_allowed_networks allow anyone from any IP to access both the nodes via SSH and the API server with a Kubernetes client. With a production cluster you may want to restrict access to specific networks
- I am setting schedule_workloads_on_masters to false here because I prefer running my workloads on worker nodes and leave the masters for Kubernetes components only. If you want to save money, you can also schedule workloads on masters
- each pool here has a different location, to illustrate how easy it is to create a multi region cluster. Each node pool simply requires a unique name, the location, and either a fixed instance count, or - if autoscaling is enabled - both the minimum and the maximum number of nodes to autoscale
- enable_encryption can in theory be disabled since the traffic between the nodes goes through a private network, but I like to enable it just as an additional security improvement even if it adds a little overhead

As you can see, the required configuration is very small and very easy to understand, and is all that you need to create a cluster besides the hetzner-k3s binary, which you can download from here. Make sure the hetzner-k3s binary is available in the PATH and make it an executable; I like to rename it to hetzner-k3s to save some typing.

If you are on a Mac, it's even easier if you install it with Homebrew:

brew install vitobotta/tap/hetzner_k3s

I will add support for Linux too to the Homebrew formula.

So, to create a cluster, make sure you have kubectl installed and available in your PATH, then just run

hetzner-k3s create --config cluster.yaml

You will see some output describing what resources the tool is creating, and at the end a file named kubeconfig will be created in your current directory (unless you have specified a different path and filename in the config file). The process should take just a few minutes.

Now, to interact the cluster, run

export KUBECONFIG=kubeconfig
kubectl get nodes

to see the list of nodes. That's it! You now have a fully functional cluster with autoscaling that allows you to provision load balancers and persistent volumes out of the box. 

Next steps

The tool doesn't install additional software in the Kubernetes cluster because this would be way too opinionated. For example people like different ingress controllers, different monitoring systems, etc. Therefore it's up to you what you do with the cluster once it's created. However, if you are just getting started with Kubernetes in general, take a look at this tutorial written by a contributor to see an example of how to install things like cert-manager (to provision TLS certificates for free using Let's Encrypt) and the Nginx ingress controller.

For other cluster operations such as upgrades, refer to the project's README.


As of today, I believe hetzner-k3s is the easiest and quickest way to create a fully functional Kubernetes cluster in Hetzner Cloud from scratch, completely automated and with a very simple and small config file. Other tools may offer more configuration options, but for most people the configuration available with hetzner-k3s will be suffiicient.

If you are a Kubernetes user and are looking to create a super cheap but fully functional cluster as easily and quickly as possible, give this tool a try! Hetzner Cloud now has locations in the USA too, which is awesome. If you do try the tool, let me know in the comments what you think. If you run into a problem using it, then open an issue on Github and I will do my best to help.  For general questions about Kubernetes not specific to my tool, feel free to open a discussion here.

© Vito Botta