How to deploy containerd with kata containers?

Page content

In this post I will show you how you can install and use kata-container engine in kubernetes.

Parst of the K8S Security series

What is Kata container engine

Kata Containers is an open source community working to build a secure container runtime with lightweight virtual machines that feel and perform like containers, but provide stronger workload isolation using hardware virtualization technology as a second layer of defense. (Source: Kata Containers Website )

Kata container engine

Enable qvemu

I will use Vagrant and VirtualBox for running the AlmaLinux VM so first I need to enable then Nested virtualization on the VM:

VBoxManage modifyvm alma8 --nested-hw-virt on

After the Linux is booted test the virtualization flag in the VM:

egrep --color -i "svm|vmx" /proc/cpuinfo

If you find one of this flags everything is ok. Now we need to enable the kvm kernel module.

sudo modprobe kvm-intel

Install Kata container engine

cat <<EOF | sudo -E tee /etc/yum.repos.d/kata-containers.repo
[kata-containers]
name=Kata Containers
baseurl=http://mirror.centos.org/centos/$releasever/virt/$basearch/kata-containers
enabled=1
gpgcheck=0
skip_if_unavailable=1
EOF

cat <<EOF | sudo -E tee /etc/yum.repos.d/advanced-virtualization.repo
[advanced-virtualization]  
name=Advanced Virtualization from CentOS $releasever
baseurl=http://mirror.centos.org/centos/$releasever/virt/$basearch/advanced-virtualization
enabled=1
gpgcheck=0
skip_if_unavailable=1
EOF

sudo dnf install epel-release nano wget
sudo dnf module disable virt
sudo -E dnf install -y kata-containers

Install and configure containerd

First I install containerd then I add Kata container as a containerd plugin to the config.

sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y containerd.io

sudo mkdir -p /etc/containerd
sudo containerd config default > /etc/containerd/config.toml

nano /etc/containerd/config.toml
...
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
          runtime_type = "io.containerd.kata.v2"

# Restart containerd
sudo systemctl restart containerd
systemctl enable containerd.service

echo "runtime-endpoint: unix:///run/containerd/containerd.sock" > /etc/crictl.yaml
crictl ps

Now I can start a Kata container from commadnline.

sudo ctr image pull docker.io/library/busybox:latest
sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh

Install nerdctl

I like to use nerdctl instad of ctr or crictl cli so I will install it.

wget https://github.com/containerd/nerdctl/releases/download/v0.11.0/nerdctl-0.11.0-linux-amd64.tar.gz

tar -xzf nerdctl-0.11.0-linux-amd64.tar.gz
mv nerdctl /usr/local/bin
nerdctl ps

Install Kubernetes

Configure Kernel parameters for Kubernetes.

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
kvm-intel
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system

Disable swap for Kubernetes.

free -h
swapoff -a
swapoff -a
sed -i.bak -r 's/(.+ swap .+)/#\1/' /etc/fstab
free -h

The I will add the kubernetes repo and Install the packages.

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

dnf install kubelet kubeadm kubectl -y

Start Kubernetes with containerd engine.

export IP=172.17.13.10

dnf install -y iproute-tc

systemctl enable kubelet.service

echo "KUBELET_EXTRA_ARGS=--cgroup-driver=systemd" | tee /etc/sysconfig/kubelet

kubeadm config images pull --cri-socket=unix:///run/containerd/containerd.sock
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=$IP --cri-socket=unix:///run/containerd/containerd.sock

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl get no
nerdctl -n k8s.io ps
crictl ps

kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-

Inincialize network

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl aplly -f kube-flannel.yml

OR

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
wget https://docs.projectcalico.org/manifests/custom-resources.yaml

nano custom-resources.yaml
...
      cidr: 10.244.0.0/16
...

kubectl apply -f custom-resources.yaml

Start Deployment

First I create a RuntimeClass for Kata then start a pod with this RuntimeClass.

cat<<EOF | kubectl apply -f -
kind: RuntimeClass
apiVersion: node.k8s.io/v1
metadata:
    name: kata
handler: kata
EOF

cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: untrusted
  name: www-kata
spec:
  runtimeClassName: kata
  containers:
  - image: nginx:1.18
    name: www
    ports:
    - containerPort: 80
EOF

$ kubectl get po
NAME       READY   STATUS    RESTARTS   AGE
www-kata   1/1     Running   0          2m47s


$ kubectl describe po www-kata
...
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  2m42s  default-scheduler  Successfully assigned default/www-kata to alma8
  Normal  Pulled     2m13s  kubelet            Container image "nginx:1.18" already present on machine
  Normal  Created    2m13s  kubelet            Created container www
  Normal  Started    2m11s  kubelet            Started container www