CNI with Multus

Multus is a CNI plugin for Kubernetes which enables attaching multiple network interfaces to pods. Multus support for Charmed Kubernetes is provided by the Multus charm, which must be deployed into a Kubernetes model in Juju.

Requirements

Juju 2.8.0

The Multus charm requires Juju 2.8.0 or newer.

CNI providers

Multus is not a replacement for other CNI providers. Your Charmed Kubernetes deployment must include at least one of the base CNI providers documented in the CNI overview page.

Persistent volume support

In order to deploy Multus, you will need a Kubernetes model in Juju, which requires persistent volume support to be enabled in your Charmed Kubernetes cluster.

If your cluster includes any of the cloud integrator charms, then you should have persistent volume support already. Otherwise, you can read the Storage documentation page to learn how to enable persistent volume support by adding Ceph to your cluster.

Creating a Kubernetes model in Juju

To deploy the Multus charm, you will first need a Kubernetes model in Juju.

Make sure your local kubeconfig is pointing to the correct Kubernetes cluster:

juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config

Next, add your Kubernetes as a cloud to your Juju controller:

juju add-k8s my-k8s-cloud --controller $(juju switch | cut -d: -f1)

And create a new Kubernetes model:

juju add-model my-k8s-model my-k8s-cloud

Deploying Multus

Once all of the requirements have been met, you can deploy Multus into a Kubernetes model by running:

juju deploy multus

Configuring the Default CNI

Multus delegates to a single CNI network by default. If you have multiple CNI subordinates in your cluster then you can use the default-cni config on kubernetes-control-plane to set the default:

juju config kubernetes-control-plane default-cni=flannel

Multus will use this as the default network as well.

Example: Pod with multiple Flannel interfaces

If you have Flannel in your cluster, then you can use it to attach additional network interfaces to your pods. In this example, we'll create an Ubuntu pod with 2 extra Flannel interfaces, in addition to the default one.

Create a NetworkAttachmentDefinition for Flannel:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: flannel
spec:
  config: |
    {
        "cniVersion": "0.3.1",
        "plugins": [
          {
            "type": "flannel",
            "delegate": {
              "hairpinMode": true,
              "isDefaultGateway": true
            }
          },
          {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
          }
        ]
    }

Create a Pod with the k8s.v1.cni.cncf.io/networks annotation:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  annotations:
    k8s.v1.cni.cncf.io/networks: flannel, flannel
spec:
  containers:
  - name: ubuntu
    image: ubuntu
    command: ['sleep', '3600']

That's it. After the pod comes up, you can exec into the pod and confirm that it has multiple network interfaces with IP addresses belonging to the Flannel network:

$ kubectl exec -it ubuntu bash
root@ubuntu:/# apt update && apt install -y net-tools
...
root@ubuntu:/# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 8951
        inet 10.1.34.6  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::c001:5aff:feb8:fefd  prefixlen 64  scopeid 0x20<link>
        ether c2:01:5a:b8:fe:fd  txqueuelen 0  (Ethernet)
        RX packets 5447  bytes 18185162 (18.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4666  bytes 356784 (356.7 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

net1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 8951
        inet 10.1.34.3  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::78fe:9ff:fe80:ec57  prefixlen 64  scopeid 0x20<link>
        ether 7a:fe:09:80:ec:57  txqueuelen 0  (Ethernet)
        RX packets 42  bytes 3220 (3.2 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11  bytes 838 (838.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

net2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 8951
        inet 10.1.34.4  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::acad:f6ff:feaa:416f  prefixlen 64  scopeid 0x20<link>
        ether ae:ad:f6:aa:41:6f  txqueuelen 0  (Ethernet)
        RX packets 40  bytes 3088 (3.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11  bytes 838 (838.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

In this case, eth0 is the default interface that gets attached to all pods, while net1 and net2 are the additional interfaces that Multus attached to the pod based on the pod's k8s.v1.cni.cncf.io/networks annotation.

For additional examples of how you can use Multus, please refer to the official Multus documentation here: Create network attachment definition.

Troubleshooting

If there is an issue with Multus, it can be useful to inspect the Juju logs. To see a complete set of logs for Multus:

juju debug-log --replay --include=multus

For additional troubleshooting pointers, please see the dedicated troubleshooting page.