How to deploy containerd with kata containers?

August 20, 2021

Kubernetes

Page content

In this post I will show you how you can install and use kata-container engine in kubernetes.

Parts of the K8S Security Lab series

What is Kata container engine

Kata Containers is an open source community working to build a secure container runtime with lightweight virtual machines that feel and perform like containers, but provide stronger workload isolation using hardware virtualization technology as a second layer of defense. (Source: Kata Containers Website )

Kata container engine

Enable qvemu

I will use Vagrant and VirtualBox for running the AlmaLinux VM so first I need to enable then Nested virtualization on the VM:

VBoxManage modifyvm alma8 --nested-hw-virt on

After the Linux is booted test the virtualization flag in the VM:

egrep --color -i "svm|vmx" /proc/cpuinfo

If you find one of this flags everything is ok. Now we need to enable the kvm kernel module.

sudo modprobe kvm-intel

Install Kata container engine

I use Almalinux 8 for this install:

sudo -E dnf install -y centos-release-advanced-virtualization
sudo -E dnf module disable -y virt:rhel

cat <<EOF | sudo -E tee /etc/yum.repos.d/kata-containers.repo
[kata-containers]
name=Kata Containers
baseurl=http://mirror.centos.org/centos-8/8/virt/x86_64/kata-containers
enabled=1
gpgcheck=1
skip_if_unavailable=1
EOF

sudo -E dnf install -y kata-containers

Install and configure containerd

First I install containerd then I add Kata container as a containerd plugin to the config.

sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y containerd.io

sudo mkdir -p /etc/containerd
sudo containerd config default > /etc/containerd/config.toml

nano /etc/containerd/config.toml
...
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
          runtime_type = "io.containerd.kata.v2"

# Restart containerd
sudo systemctl restart containerd
systemctl enable containerd.service

echo "runtime-endpoint: unix:///run/containerd/containerd.sock" > /etc/crictl.yaml
crictl ps

Now I can start a Kata container from commadnline.

sudo ctr image pull docker.io/library/busybox:latest
sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh

Install nerdctl

I like to use nerdctl instad of ctr or crictl cli so I will install it.

wget https://github.com/containerd/nerdctl/releases/download/v0.11.0/nerdctl-0.11.0-linux-amd64.tar.gz

tar -xzf nerdctl-0.11.0-linux-amd64.tar.gz
mv nerdctl /usr/local/bin
nerdctl ps

Install Kubernetes

Configure Kernel parameters for Kubernetes.

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
kvm-intel
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system

Disable swap for Kubernetes.

free -h
swapoff -a
swapoff -a
sed -i.bak -r 's/(.+ swap .+)/#\1/' /etc/fstab
free -h

The I will add the kubernetes repo and Install the packages.

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

dnf install kubelet kubeadm kubectl -y

Start Kubernetes with containerd engine.

export IP=172.17.13.10

dnf install -y iproute-tc

systemctl enable kubelet.service

echo "KUBELET_EXTRA_ARGS=--cgroup-driver=systemd" | tee /etc/sysconfig/kubelet

kubeadm config images pull --cri-socket=unix:///run/containerd/containerd.sock
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=$IP --cri-socket=unix:///run/containerd/containerd.sock

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl get no
nerdctl -n k8s.io ps
crictl ps

kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-

Inincialize network

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl aplly -f kube-flannel.yml

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
wget https://docs.projectcalico.org/manifests/custom-resources.yaml

nano custom-resources.yaml
...
      cidr: 10.244.0.0/16
...

kubectl apply -f custom-resources.yaml

Start Deployment

First I create a RuntimeClass for Kata then start a pod with this RuntimeClass.

cat<<EOF | kubectl apply -f -
kind: RuntimeClass
apiVersion: node.k8s.io/v1
metadata:
    name: kata
handler: kata
EOF

cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: untrusted
  name: www-kata
spec:
  runtimeClassName: kata
  containers:
  - image: nginx:1.18
    name: www
    ports:
    - containerPort: 80
EOF

$ kubectl get po
NAME       READY   STATUS    RESTARTS   AGE
www-kata   1/1     Running   0          2m47s


$ kubectl describe po www-kata
...
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  2m42s  default-scheduler  Successfully assigned default/www-kata to alma8
  Normal  Pulled     2m13s  kubelet            Container image "nginx:1.18" already present on machine
  Normal  Created    2m13s  kubelet            Created container www
  Normal  Started    2m11s  kubelet            Started container www

Parts of the K8S Security Lab series

Container Runetime Security

Advanced Kernel Security

Network Security

Secure Kubernetes Install

User Security

Image Security

Pod Security

Secret Security

Monitoring and Observability

Backup