Self-healing Kubernetes deployments with MicroK8s and Portainer

In this tutorial, we will install Ubuntu and MicroK8s on all four of the Raspberry Pi nodes. Then we will enable Portainer, an opinionated graphical user interface for Kubernetes cluster management, which we will use to spin up our containers. After setting this up, we will try and power down one of the nodes to see that our workload will survive.

What you’ll learn

• Install Ubuntu on the Raspberry Pi
• Install and configure MicroK8s
• Configure local DNS server
• Setup testing domain
• Deploy NGINX

What you’ll need

• 4 Raspberry Pi’s model 4 with 8GB RAM and at least 32GB storage
• PoE switch with 4 ports and 4 Raspberry Pi HATs or 4 standart Raspberry Pi power supplies

Writing Ubuntu on MicroSD card

Let’s start by installing Ubuntu Server 20.04. For more detailed instructions you can follow the tutorial on how to install Ubuntu on Raspberry Pi.
First, insert the microSD card into your computer. Then you will need to install PRI imager by following one of these links or if you’re on ubuntu by running:

sudo snap install rpi-imager

Once this is done, start the Imager and open the “CHOOSE OS” menu:

Scroll down the menu click “Other general-purpose OS”:

Here you you can select Ubuntu and see a list of download options:

For this tutorial we recommend Ubuntu Server 20.04 64-bit download:

Select the image and open the “Choose storage” menu. Select the microSD card you have inserted:

And finally, click write and wait a few minutes.

Changing initial config files

Next, we need to configure the WiFi. With the microSD card still inserted in your laptop, open a file manager and locate the “writable” partition on the card. Find /etc/netplan/ And create the following network.yaml file replacing the values with your network credentials:

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      optional: true
  wifis:
    wlan0:
      dhcp4: true
      optional: true
      access-points:
        "YOUR_WIFI_NAME":
          password: "YOUR_PASSWORD"

Save the file, copy it somewhere to use with other Raspberries.
The next step is to disable network configuration done by cloud init. Find /etc/cloud/cloud.d/ and create the following 99-network-config.yaml file:

network:
  config: disabled

After this we need to edit the boot parameters for MicroK8s to work properly. Open cmdline.txt and add the following without changing the other default parameters:

cgroup_enable=memory cgroup_memory=1

Finally, extract the card from your laptop and put it in the Raspberry Pi.
After waiting a couple of minutes for the Raspberry to start, check your local network devices for its IP address. If no IPs are showing, you may need to login and reboot your Raspberry Pi with a USB keyboard and monitor connected to it, in order to check and change the network settings manually . If the network is not working, you can check /etc/netplan/network.yaml to see if all the information is correct. If not, edit the file and do:

sudo netplan apply

Now you should be able to connect to the Raspberry Pi. Repeat this process for the other nodes.

Before we install MicroK8s we need to Change hostnames for every node, so they would differ and MicroK8s would recognise them properly. Connect to the node via ssh and open /etc/hostname:

ssh ubuntu@<IP address>
sudo bash -c "cat > ./root/preseeder.sh" << EOF
ubuntu1
EOF

Now we can install MicroK8s, join the user in the MicroK8s group and gain access to .kube caching directory:

sudo snap install microk8s --classic --channel=1.26
sudo usermod -a -G microk8s ubuntu
sudo chown -f -R ubuntu ~/.kube"

Repeat the MicroK8s installation process on the other nodes.

MicroK8s uses a namespaced kubectl command to prevent conflicts with any existing installs of kubectl. If you don’t have an existing install, it is easier to add an alias (append to ~/.bash_aliases) like this:

echo "alias kubectl='microk8s kubectl'" >> ~/.bash_aliases
source .bash_aliases

Enable MicroK8s high availability

In order to enable high availability (HA) we will need to generate 3 join tokens and run the join command on the other MicroK8s machines. On the initial node, run:

microk8s add-node

This will output a command with a generated token such as:

microk8s join 10.128.63.86:25000/567a21bdfc9a64738ef4b3286b2b8a69

Copy this command and run it from the next node. It may take a few minutes to successfully join.
Repeat this process (generate a token, run it from the joining node) for the third and forth nodes.

Enable Add-ons

Now Let’s enable DNS and storage add-ons plus the Portainer add-on:

microk8s enable dns 
microk8s enable storage 
microk8s enable portainer

We can now navigate to Nodeport service on port 30777 to login to the Portainer dashboard. For the first login we are prompted to create a user and after we do we can use the different features of Portainer:

In order for us to see how our workload is being distributed we need to use a DNS server. In this example we will set up a simple local DNS Server using bind9. With the DNS server configured, we can then use an ingress controller which will load balance across each node.
Access /etc/hosts and add every node IP and name to it:

sudo echo "<IP1> ubuntu1" >> /etc/hosts
sudo echo "<IP2> ubuntu2" >> /etc/hosts
sudo echo "<IP3> ubuntu3" >> /etc/hosts
sudo echo "<IP4> ubuntu4" >> /etc/hosts

Next install bind9 dns server and edit the firewall rules to allow it:

sudo apt install bind9 -y
sudo ufw allow Bind9

Then we need to edit the config options for the server. It should look like this:

sudo vim /etc/bind/named.conf.options

options {
        directory "/var/cache/bind";
 
        dnssec-validation auto;
 
        listen-on-v6 { any; };
        listen-on { any; };
        allow-query { any; };
        forwarders {
            8.8.8.8;
            8.8.4.4;
        };
};

Check for errors, restart the service and enable it:

sudo named-checkconf
sudo systemctl restart named
sudo systemctl enable named

After that test that external connectivity resolves with:

nslookup google.com <NODE IP>

Setup testing domain

Now we will create microk8s.testing domain. Edit the config like this:

sudo vi /etc/bind/named.conf.local
zone "microk8s.testing" {
   type master;
   file "/etc/bind/db.microk8s.testing";
};

Create the database file:

sudo touch /etc/bind/db.microk8s.testing

And add the following to it replacing the values with your nodes IP addresses:

sudo vim /etc/bind/db.microk8s.testing 
;
; BIND data file for local loopback interface
;
$TTL	604800
@	IN	SOA	ns1.microk8s.testing. microk8s.testing. (
			      2		; Serial
			  300		; Refresh
			  300		; Retry
			  300		; Expire
			  300 )	; Negative Cache TTL
;
@       IN	    NS      microk8s.testing.
@       IN	    A       <NODE 1>        
@       IN      A       <NODE 2>       
@       IN      A       <NODE 3>
@       IN      A       <NODE 4> 

Now reload everything:

systemctl reload bind9
sudo rndc reload

And check if dns resolves:

nslookup microk8s.testing <NODE IP>

Now we will need to set all the machines to use the DNS server we configured. First it is recommended to set static IPs for the machines on your router. Now let’s change the netplan to use our DNS, it should look like this (notice that we’re setting DHCP to false):

sudo vim /etc/netplan/network.yaml

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true
      optional: true
  wifis:
    wlan0:
      dhcp4: false
      optional: true
      addresses: [<NODE IP>/24]
      gateway4: <NETWORK GATEWAY>
      nameservers:
        addresses: [<DNS SERVER NODE IP>]
      access-points:
        "YOUR_WIFI_NAME":
          password: "YOUR_PASSWORD"

sudo netplan apply

Repeat the netplan step on each node, don’t forget to change the address of the node.
Now we can reconfigure the MicroK8s dns add-on to this server. First - disable the add on and then enable it with specifying our servers IP:

microk8s disable dns
microk8s enable dns:<DNS ADDRESS>

Now the cluster is ready for action. In order to deploy a sample application, such as nginx, we can go to the Portainer UI. Select applications, create from manifest and create an nginx deployment, service and ingress.

apiVersion: apps/v1 
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 4
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: ""
        name: nginx
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          timeoutSeconds: 30
        resources: {}
      restartPolicy: Always
      serviceAccountName: ""
status: {}
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: NodePort
  selector:
     app: nginx
  ports:
  - name: nginx
    protocol: TCP
    nodePort: 30080
    port: 80
    targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
spec:
  rules:
  - host: "microk8s.testing"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port: 
              number: 80
  ingressClassName: public

Once deployed you can sshuttle to any node and check it:

sshuttle -r ubuntu@<NODE IP> 0.0.0.0/0 --dns

Now you can turn off any node except the one with your DNS server and see how your workload will survive.

We have covered installing Ubuntu on Raspberry Pi’s, deploying MicroK8s and Portainer on Ubuntu, setting up a DNS server and creating a simple NGINX deployment to see how our edge cluster will keep its workload healthy.

• Visit the MicroK8s website
• Follow more MicroK8s tutorials
• Learn more about Canonical Kubernetes solutions
• Get in touch about your Kubernetes use cases
• Manage and deploy containers in your environment quickly and easily with Portainer