How to setup Headscale Server and Headscale UI on Kubernetes

Why would you want headscale server on kubernetes? For high availability. There are two ways to do it:

1. You have multiple replicas of headscale server and persistent volume claim so that your headscale server is always up.
2. You have 1 replica and you have the data saved on external drive.

In this article I will show you how to do #2. The reason being is that, in my particular case, I almost always have to reset my entire kubernetes cluster for some reason. And if I do that all the time, I lose the data from headscale that connects all the different machine. So thats why I like number 2 for my case. Whenever I need the old data if I have to restart my entire kubernetes cluster from the ground up, my previous headscale will always be there.

Here are the overview steps:

1. Compose the Headscale Docker Image and push it into your private Docker Hub repo. You need to do this because whenever kubernetes build the headscale image it needs to know where to pull it from.
2. Know which machine that your external hard drive is connected to. Mount that external hard drive to that machine. Again, the reason why I am putting it in an external hard drive is because if I have to destroy and restart my nodes for whatever reason, the prior headscale data will always be there. If you want a different setup, then I would recommend presistent storage with Longhorn (but we will not cover that in this article).
3. Open up the ports for that machine so that Headscale can operate properly.
4. Copy and paste the yaml files
5. Headscale working!

Step 1: Compose Docker Image

Here is the Dockerfile

FROM alpine:3.17.1

# ---
# upgrade system and installed dependencies for security patches
RUN --mount=type=cache,sharing=private,target=/var/cache/apk \
    set -eux; \
    apk upgrade

# ---
# copy headscale
RUN --mount=type=cache,target=/var/cache/apk \
    --mount=type=tmpfs,target=/tmp \
    set -eux; \
    cd /tmp; \
    { \
        export \
            HEADSCALE_VERSION=0.20.0 \
            HEADSCALE_SHA256=44214cc639881d546efee80e3f224395d19fde232f764179ded8fe4b5025674b; \
        wget -q -O headscale https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_arm64; \
        echo "${HEADSCALE_SHA256} *headscale" | sha256sum -c - >/dev/null 2>&1; \
        chmod +x headscale; \
        mv headscale /usr/local/bin/; \
    }; \
    # smoke tests
    [ "$(command -v headscale)" = '/usr/local/bin/headscale' ]; \
    headscale version

Take a look at Line 19:

Line 19: headscale https://github.com/juanfont/headscale/releases/download/v${HEADSCALE_VERSION}/headscale_${HEADSCALE_VERSION}_linux_arm64;

Notice at the very end it says

Line 19: ... ${HEADSCALE_VERSION}_linux_arm64;

If your machine is different, make sure to change it to the right version for your machine.

${HEADSCALE_VERSION}_linux_arm64;
${HEADSCALE_VERSION}_linux_amd64;

If you do change it, you will also have to change the checksum, since that headscale is checking to see if it is the current and valid github url. You can get the checksum for your machine on the headscale github page under checksums.txt: https://github.com/juanfont/headscale/releases/. Line 18 of the above Dockerfile is the checksum, the one that says SHA256.

So change it to amd, if you have an amd machine, or arm if you have arm.

Copy that Dockerfile and put it into your machine

sudo mkdir -p /headscale_config
sudo nano /headscale_config/Dockerfile

Copy and paste the Dockerfile, then CTRL+X and Y to save.

Go to hub.docker.com. Create an account if dont have one already. Then build the image and push it.

cd /headscale_config
docker login -u YOUR_DOCKERHUB_USERNAME -p YOUR_DOCKERHUB_PASSWORD
docker build -t YOUR_DOCKERHUB_USERNAME/headscale:0.20.0 .
docker push YOUR_DOCKERHUB_USERNAME/headscale:0.20.0
cd ../

Make sure to replace YOUR_DOCKERHUB_USERNAME and YOUR_DOCKERHUB_PASSWORD to your own dockerhub username and dockerhub password.

Step 2 & 3: Mount your external hard drive to one of your nodes

lsblk

It should look something similar to that. My external hard drive main partition is on sda2. Your may be different, just look for the size so you can tell which is your external hard drive. Since my external hard drive is 8TB, i know that it is the sda2. Notice under the size column it says 7.3T. Notice that my hard drive is on knode3, my 3rd machine.

Run the following commands. This opens the ports and mounts the external drive to your machine.

sudo ufw allow 47239/tcp
sudo ufw allow 9443/tcp
sudo systemctl restart ufw
rm -rf /mnt/external_harddrive
sudo mkdir /mnt/external_harddrive
sudo mkdir /mnt/external_harddrive/etc/headscale
sudo mkdir /mnt/external_harddrive/var/lib/headscale
sudo mount /dev/sda2 /mnt/external_harddrive

Step 4: Copy and Paste Yaml Files

kubectl create namespace headscale

kubectl create secret docker-registry dockerhub-secret \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=YOUR_DOCKERHUB_USERNAME \
  --docker-password=YOUR_DOCKERHUB_PASSWORD \
  --docker-email=YOUR_EMAIL \
  -n headscale

kubectl delete -f /kubernetes_config/headscale/headscale_server_config.yaml
mkdir -p /kubernetes_config/headscale
rm -rf /kubernetes_config/headscale/headscale_server_config.yaml
nano /kubernetes_config/headscale/headscale_server_config.yaml

This is the deployment yaml file. Lets call it headscale_server_config.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: headscale
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: headscale
  namespace: headscale
spec:
  replicas: 1
  selector:
    matchLabels:
      app: headscale
  template:
    metadata:
      labels:
        app: headscale
    spec:
      initContainers:
      - name: init-hostpath
        image: alpine
        command: ["sh", "-c", "mkdir -p /mnt/external_harddrive/var/lib/headscale && mkdir -p /var/lib/headscale && mkdir -p /mnt/external_harddrive/etc/headscale && mkdir -p /etc/headscale"]
        volumeMounts:
        - name: headscale-data
          mountPath: /mnt/external_harddrive/var/lib/headscale
      containers:
      - name: headscale
        image: YOUR_DOCKERHUB_USERNAME/headscale:0.20.0
        command: ["headscale", "serve"]
        ports:
        - containerPort: 47239
        volumeMounts:
        - name: headscale-config
          mountPath: /etc/headscale
        - name: headscale-data
          mountPath: /mnt/external_harddrive/var/lib/headscale
      - name: headscale-ui
        image: ghcr.io/gurucomputing/headscale-ui:latest
        ports:
        - containerPort: 443
        volumeMounts:
        - name: headscale-config
          mountPath: /etc/headscale
        - name: headscale-data
          mountPath: /mnt/external_harddrive/var/lib/headscale
      imagePullSecrets:
      - name: dockerhub-secret
      volumes:
      - name: headscale-config
        configMap:
          name: headscale-config
      - name: headscale-data
        hostPath:
          path: /mnt/external_harddrive/var/lib/headscale
      nodeSelector:
        kubernetes.io/hostname: knode3
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: headscale-config
  namespace: headscale
data:
  config.yaml: |
    server_url: https://YOUR_DOMAIN.com:443
    listen_addr: 0.0.0.0:47239
    metrics_listen_addr: 127.0.0.1:9090
    private_key_path: /mnt/external_harddrive/var/lib/headscale/private.key
    noise:
      private_key_path: /mnt/external_harddrive/var/lib/headscale/noise_private.key
    ip_prefixes:
      - 100.64.0.0/10
    disable_check_updates: true
    db_type: sqlite3
    db_path: /mnt/external_harddrive/var/lib/headscale/db.sqlite
    log:
      format: text
      level: info
    dns_config:
      override_local_dns: true
      nameservers:
        - 1.1.1.1
        - 1.0.0.1
      magic_dns: true
      base_domain: YOUR_DOMAIN.com
    logtail:
      enabled: false
---
apiVersion: v1
kind: Service
metadata:
  name: headscale
  namespace: headscale
spec:
  selector:
    app: headscale
  ports:
    - name: headscale-port
      protocol: TCP
      port: 47239
      targetPort: 47239
    - name: headscale-ui-port
      protocol: TCP
      port: 9443
      targetPort: 443
  type: LoadBalancer

IMPORTANT — CHANGE THESE LINES TO YOUR OWN MACHINE:

In the yaml file, change these to match your machine

Line 30: image: YOUR_DOCKERHUB_USERNAME/headscale:0.20.0
Line 58: kubernetes.io/hostname: knode3
Line 67: server_url: https://headscale-connection.YOUR_DOMAIN.com:443
Line 87: base_domain: YOUR_DOMAIN.com

Line 30: YOUR_DOCKERHUB_USERNAME to your docker username
Line 58: knode3 to whatever your machine name is
Line 67: YOUR_DOMAIN to your domain name
Line 87: YOUR_DOMAIN to your domain name

kubectl apply -f /kubernetes_config/headscale/headscale_server_config.yaml

Your machine now has headscale and headscale ui!

You can shell into the headscale pod to add nodes by typing this command:

kubectl exec -it -n headscale $(kubectl get pod -n headscale -o name) -- /bin/sh