Find out what high availability means for Kubernetes and K3s! You can turn HA on with the flip of a switch, but there is more to it if you want a truly fault-tolerant system.
What is High Availability (HA)?
In cloud computing, high availability refers to a system that can withstand the failure of one or more of its parts. Kubernetes achieves this by having three or more nodes make up the control plane. When one node fails, a new one is elected leader and functions as the primary controller. The cluster should continue to function as a whole, even with the failure of a controller.
To set up K3s in HA you need to create a primary controller and then join secondary controllers to it. If you've installed K3s before, you know how easy it is! The primary controller installation remains the same as a single-node setup. The secondary controllers are installed in a very similar way, with the addition of a flag that tells the other nodes where the primary node is located. These steps are laid out for you in the K3s documentation: High Availability with Embedded DB.
# To get started, first launch a server node with the cluster-init flag to enable clustering and a token that will be used as a shared secret to join additional servers to the cluster.
curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --cluster-init
# After launching the first server, join the second and third servers to the cluster using the shared secret
curl -sfL https://get.k3s.io | K3S_TOKEN=SECRET sh -s - server --server https://<ip or hostname of server1>:6443
At this point, your cluster is technically highly available. If a controller goes down, workloads will continue to function and new workloads can be scheduled.
What about Ingress?
Congratulations, your cluster is now resistant to failure! There's just one catch, what about ingress? Even though your cluster is HA you could still see downtime for apps that you are exposing to the internet.
K3s comes packaged with ServiceLB - formerly known as Klipper Load Balancer. This exposes any service of type LoadBalancer to the network outside your cluster.
ℹ️ If you're more comfortable with MetalLB you can swap that in for ServiceLB. If you're new to networking we recommend sticking to the defaults.
The next layer in your network depends on your setup. You will need to configure your router to forward incoming traffic to the LoadBalancer's external IP. The problem with this is if you specify a single IP, and that IP goes down, now your ingress will be broken! Our solution to this is to add a physical load balancer between our router and the cluster. We installed HAProxy on an additional Raspberry Pi and configured it to forward traffic from 80 and 443 to the external IP of the cluster. This way nodes will be able to go down without affecting our ingress.
Links
- Rancher K3s HA Installation: https://rancher.com/docs/k3s/latest/en/installation/ha/
- Klipper ServiceLB: https://docs.k3s.io/networking#service-load-balancer
- Kubernetes LoadBalancer service type: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
- MetalLB: https://metallb.org/
- HAProxy: https://www.haproxy.org/