How to deploy containerd with kata containers?
In this post I will show you how you can install and use kata-container engine in kubernetes.
Parst of the K8S Security series
- Part1: Best Practices to keeping Kubernetes Clusters Secure
- Part2: Kubernetes Hardening Guide with CIS 1.6 Benchmark
- Part3: RKE2 The Secure Kubernetes Engine
- Part4: RKE2 Install With cilium
- Part5: Kubernetes Certificate Rotation
- Part6: Hardening Kubernetes with seccomp
- Part7a: RKE2 Pod Security Policy
- Part7b: Kubernetes Pod Security Admission
- Part7c: Pod Security Standards using Kyverno
- Part8: Kubernetes Network Policy
- Part9: Kubernetes Cluster Policy with Kyverno
- Part10: Using Admission Controllers
- Part11a: Image security Admission Controller
- Part11b: Image security Admission Controller V2
- Part11c: Image security Admission Controller V3
- Part12: Continuous Image security
- Part13: K8S Logging And Monitoring
- Part14: Kubernetes audit logs and Falco
- Part15a Image Signature Verification with Connaisseur
- Part15b Image Signature Verification with Connaisseur 2.0
- Part15c Image Signature Verification with Kyverno
- Part16a Backup your Kubernetes Cluster
- Part16b How to Backup Kubernetes to git?
- Part17a Kubernetes and Vault integration
- Part17b Kubernetes External Vault integration
- Part18a: ArgoCD and kubeseal to encript secrets
- Part18b: Flux2 and kubeseal to encrypt secrets
- Part18c: Flux2 and Mozilla SOPS to encrypt secrets
- Part19: ArgoCD auto image updater
- Part20: Secure k3s with gVisor
- Part21: How to use imagePullSecrets cluster-wide??
- Part22: Automatically change registry in pod definition
What is Kata container engine
Kata Containers is an open source community working to build a secure container runtime with lightweight virtual machines that feel and perform like containers, but provide stronger workload isolation using hardware virtualization technology as a second layer of defense. (Source: Kata Containers Website )
Enable qvemu
I will use Vagrant and VirtualBox for running the AlmaLinux VM so first I need to enable then Nested virtualization on the VM:
VBoxManage modifyvm alma8 --nested-hw-virt on
After the Linux is booted test the virtualization flag in the VM:
egrep --color -i "svm|vmx" /proc/cpuinfo
If you find one of this flags everything is ok. Now we need to enable the kvm kernel module.
sudo modprobe kvm-intel
Install Kata container engine
cat <<EOF | sudo -E tee /etc/yum.repos.d/kata-containers.repo
[kata-containers]
name=Kata Containers
baseurl=http://mirror.centos.org/centos/$releasever/virt/$basearch/kata-containers
enabled=1
gpgcheck=0
skip_if_unavailable=1
EOF
cat <<EOF | sudo -E tee /etc/yum.repos.d/advanced-virtualization.repo
[advanced-virtualization]
name=Advanced Virtualization from CentOS $releasever
baseurl=http://mirror.centos.org/centos/$releasever/virt/$basearch/advanced-virtualization
enabled=1
gpgcheck=0
skip_if_unavailable=1
EOF
sudo dnf install epel-release nano wget
sudo dnf module disable virt
sudo -E dnf install -y kata-containers
Install and configure containerd
First I install containerd then I add Kata container as a containerd plugin to the config.
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y containerd.io
sudo mkdir -p /etc/containerd
sudo containerd config default > /etc/containerd/config.toml
nano /etc/containerd/config.toml
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
runtime_type = "io.containerd.kata.v2"
# Restart containerd
sudo systemctl restart containerd
systemctl enable containerd.service
echo "runtime-endpoint: unix:///run/containerd/containerd.sock" > /etc/crictl.yaml
crictl ps
Now I can start a Kata container from commadnline.
sudo ctr image pull docker.io/library/busybox:latest
sudo ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh
Install nerdctl
I like to use nerdctl
instad of ctr
or crictl
cli so I will install it.
wget https://github.com/containerd/nerdctl/releases/download/v0.11.0/nerdctl-0.11.0-linux-amd64.tar.gz
tar -xzf nerdctl-0.11.0-linux-amd64.tar.gz
mv nerdctl /usr/local/bin
nerdctl ps
Install Kubernetes
Configure Kernel parameters for Kubernetes.
cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
kvm-intel
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sysctl --system
Disable swap for Kubernetes.
free -h
swapoff -a
swapoff -a
sed -i.bak -r 's/(.+ swap .+)/#\1/' /etc/fstab
free -h
The I will add the kubernetes repo and Install the packages.
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
dnf install kubelet kubeadm kubectl -y
Start Kubernetes with containerd engine.
export IP=172.17.13.10
dnf install -y iproute-tc
systemctl enable kubelet.service
echo "KUBELET_EXTRA_ARGS=--cgroup-driver=systemd" | tee /etc/sysconfig/kubelet
kubeadm config images pull --cri-socket=unix:///run/containerd/containerd.sock
kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=$IP --cri-socket=unix:///run/containerd/containerd.sock
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get no
nerdctl -n k8s.io ps
crictl ps
kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-
Inincialize network
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl aplly -f kube-flannel.yml
OR
kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
wget https://docs.projectcalico.org/manifests/custom-resources.yaml
nano custom-resources.yaml
...
cidr: 10.244.0.0/16
...
kubectl apply -f custom-resources.yaml
Start Deployment
First I create a RuntimeClass
for Kata then start a pod with this RuntimeClass
.
cat<<EOF | kubectl apply -f -
kind: RuntimeClass
apiVersion: node.k8s.io/v1
metadata:
name: kata
handler: kata
EOF
cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
labels:
app: untrusted
name: www-kata
spec:
runtimeClassName: kata
containers:
- image: nginx:1.18
name: www
ports:
- containerPort: 80
EOF
$ kubectl get po
NAME READY STATUS RESTARTS AGE
www-kata 1/1 Running 0 2m47s
$ kubectl describe po www-kata
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m42s default-scheduler Successfully assigned default/www-kata to alma8
Normal Pulled 2m13s kubelet Container image "nginx:1.18" already present on machine
Normal Created 2m13s kubelet Created container www
Normal Started 2m11s kubelet Started container www