How to deploy containerd with Firecracker?

Page content

In this post I will show you how you can install and use kata-container with Firecracker engine in kubernetes.

Parts of the K8S Security Lab series

Container Runetime Security
Advanced Kernel Security
Network Security
Secure Kubernetes Install
User Security
Image Security
  • Part1: Image security Admission Controller
  • Part2: Image security Admission Controller V2
  • Part3: Image security Admission Controller V3
  • Part4: Continuous Image security
  • Part5: trivy-operator 1.0
  • Part6: trivy-operator 2.1: Trivy-operator is now an Admisssion controller too!!!
  • Part7: trivy-operator 2.2: Patch release for Admisssion controller
  • Part8: trivy-operator 2.3: Patch release for Admisssion controller
  • Part8: trivy-operator 2.4: Patch release for Admisssion controller
  • Part8: trivy-operator 2.5: Patch release for Admisssion controller
  • Part9_ Image Signature Verification with Connaisseur
  • Part10: Image Signature Verification with Connaisseur 2.0
  • Part11: Image Signature Verification with Kyverno
  • Part12: How to use imagePullSecrets cluster-wide??
  • Part13: Automatically change registry in pod definition
  • Part14: ArgoCD auto image updater
    Pod Security
    Secret Security
    Monitoring and Observability
    Backup

    What is Kata container engine

    Kata Containers is an open source community working to build a secure container runtime with lightweight virtual machines that feel and perform like containers, but provide stronger workload isolation using hardware virtualization technology as a second layer of defense. (Source: Kata Containers Website )

    Kata container engine

    Why should you use Firecracker?

    Firecracker is a way to run virtual machines, but its primary goal is to be used as a container runtime interface, making it use very few resources by design.

    Enable qvemu

    I will use Vagrant and VirtualBox for running the AlmaLinux VM so first I need to enable then Nested virtualization on the VM:

    VBoxManage modifyvm alma8 --nested-hw-virt on
    

    After the Linux is booted test the virtualization flag in the VM:

    egrep --color -i "svm|vmx" /proc/cpuinfo
    

    If you find one of this flags everything is ok. Now we need to enable the kvm kernel module.

    sudo modprobe kvm-intel
    sudo modprobe vhost_vsock
    

    Install and configure containerd

    To work with firecracker the containerd must use devmapper for snapsoter plugin.

    sudo dnf install epel-release nano wget -y
    
    sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    sudo dnf install -y containerd.io lvm2
    

    First I installed the containerd. Devmapper is the only storage driver supported by Firecracker so now I create the thin-pool lvm for devmapper snapsoter:

    sudo pvcreate /dev/sdb
    sudo vgcreate containerd /dev/sdb
    sudo lvcreate --wipesignatures y -n data containerd -l 95%VG
    sudo lvcreate --wipesignatures y -n meta containerd -l 1%VG
    
    sudo lvconvert -y \
    --zero n \
    -c 512K \
    --thinpool containerd/data \
    --poolmetadata containerd/meta
    
    nano /etc/lvm/profile/ontainerd-thinpool.profile
    activation {
      thin_pool_autoextend_threshold=80
      thin_pool_autoextend_percent=20
    }
    
    sudo lvchange --metadataprofile ontainerd-thinpool containerd/data
    sudo lvchange --monitor y containerd/data
    

    And dmsetup will produce the following output:

    containerd-data	(253:2)
    containerd-data_tdata	(253:1)
    containerd-data_tmeta	(253:0)
    
    sudo mkdir -p /etc/containerd
    sudo containerd config default > /etc/containerd/config.toml
    nano /etc/containerd/config.toml
    [plugins]
      ...
      [plugins."io.containerd.grpc.v1.cri".containerd]
        snapshotter = "devmapper"
      ...
      [plugins."io.containerd.snapshotter.v1.devmapper"]
        pool_name = "containerd-data"
        base_image_size = "8GB"
        async_remove = false
    
    echo "runtime-endpoint: unix:///run/containerd/containerd.sock" > /etc/crictl.yaml
    
    # Restart containerd
    sudo systemctl restart containerd
    systemctl enable containerd.service
    

    Test if working correctly:

    systemctl restart containerd
    ctr images pull --snapshotter devmapper docker.io/library/hello-world:latest
    ctr run --snapshotter devmapper docker.io/library/hello-world:latest test
    
    $ ctr plugin ls
    TYPE                            ID                       PLATFORMS      STATUS
    io.containerd.content.v1        content                  -              ok
    io.containerd.snapshotter.v1    aufs                     linux/amd64    error
    io.containerd.snapshotter.v1    devmapper                linux/amd64    ok
    io.containerd.snapshotter.v1    native                   linux/amd64    ok
    io.containerd.snapshotter.v1    overlayfs                linux/amd64    ok
    io.containerd.snapshotter.v1    zfs                      linux/amd64    error
    io.containerd.metadata.v1       bolt                     -              ok
    io.containerd.differ.v1         walking                  linux/amd64    ok
    io.containerd.gc.v1             scheduler                -              ok
    io.containerd.service.v1        introspection-service    -              ok
    io.containerd.service.v1        containers-service       -              ok
    io.containerd.service.v1        content-service          -              ok
    io.containerd.service.v1        diff-service             -              ok
    io.containerd.service.v1        images-service           -              ok
    io.containerd.service.v1        leases-service           -              ok
    io.containerd.service.v1        namespaces-service       -              ok
    io.containerd.service.v1        snapshots-service        -              ok
    io.containerd.runtime.v1        linux                    linux/amd64    ok
    io.containerd.runtime.v2        task                     linux/amd64    ok
    io.containerd.monitor.v1        cgroups                  linux/amd64    ok
    io.containerd.service.v1        tasks-service            -              ok
    io.containerd.internal.v1       restart                  -              ok
    io.containerd.grpc.v1           containers               -              ok
    io.containerd.grpc.v1           content                  -              ok
    io.containerd.grpc.v1           diff                     -              ok
    io.containerd.grpc.v1           events                   -              ok
    io.containerd.grpc.v1           healthcheck              -              ok
    io.containerd.grpc.v1           images                   -              ok
    io.containerd.grpc.v1           leases                   -              ok
    io.containerd.grpc.v1           namespaces               -              ok
    io.containerd.internal.v1       opt                      -              ok
    io.containerd.grpc.v1           snapshots                -              ok
    io.containerd.grpc.v1           tasks                    -              ok
    io.containerd.grpc.v1           version                  -              ok
    io.containerd.grpc.v1           cri                      linux/amd64    ok
    

    Install nerdctl

    I like to use nerdctl instad of ctr or crictl cli so I will install it.

    wget https://github.com/containerd/nerdctl/releases/download/v0.11.0/nerdctl-0.11.0-linux-amd64.tar.gz
    
    tar -xzf nerdctl-0.11.0-linux-amd64.tar.gz
    mv nerdctl /usr/local/bin
    nerdctl ps
    

    Install tools

    yum install git -y
    
    sudo git clone https://github.com/ahmetb/kubectx /opt/kubectx
    sudo ln -s /opt/kubectx/kubectx /usr/local/sbin/kubectx
    sudo ln -s /opt/kubectx/kubens /usr/local/sbin/kubens
    

    Install Kubernetes

    Configure Kernel parameters for Kubernetes.

    cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
    overlay
    br_netfilter
    kvm-intel
    vhost_vsock
    EOF
    
    sudo modprobe overlay
    sudo modprobe br_netfilter
    
    cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
    net.bridge.bridge-nf-call-iptables  = 1
    net.ipv4.ip_forward                 = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    EOF
    
    sysctl --system
    

    Disable swap for Kubernetes.

    free -h
    swapoff -a
    swapoff -a
    sed -i.bak -r 's/(.+ swap .+)/#\1/' /etc/fstab
    free -h
    

    The I will add the kubernetes repo and Install the packages.

    cat <<EOF > /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
    enabled=1
    gpgcheck=1
    repo_gpgcheck=1
    gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    EOF
    
    dnf install kubelet kubeadm kubectl -y
    

    Start Kubernetes with containerd engine.

    export IP=172.17.13.10
    
    dnf install -y iproute-tc
    
    systemctl enable kubelet.service
    
    echo "KUBELET_EXTRA_ARGS=--cgroup-driver=systemd" | tee /etc/sysconfig/kubelet
    
    kubeadm config images pull --cri-socket=unix:///run/containerd/containerd.sock
    kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=$IP --cri-socket=unix:///run/containerd/containerd.sock
    
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    kubectl get no
    nerdctl -n k8s.io ps
    crictl ps
    
    kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-
    

    Inincialize network

    wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
    kubectl aplly -f kube-flannel.yml
    

    OR

    kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
    wget https://docs.projectcalico.org/manifests/custom-resources.yaml
    
    nano custom-resources.yaml
    ...
          cidr: 10.244.0.0/16
    ...
    
    kubectl apply -f custom-resources.yaml
    

    Install Kata container engine

    If all the Nodes are ready deploy a Daemonsets to build Kata containers and firecracker wit kata-deploy:

    $ kubectl get no
    
    NAME    STATUS   ROLES                  AGE     VERSION
    alma8   Ready    control-plane,master   2m31s   v1.22.1
    
    # Installing the latest image
    
    kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml
    kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml
    
    # OR
    # Installing the stable image
    
    kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml
    kubectl apply -f https://raw.githubusercontent.com/kata-containers/kata-containers/main/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy-stable.yaml
    

    Verify the pod:

    kubens kube-system
    
    $ kubectl get DaemonSet
    NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
    kata-deploy   1         1         1       1            1           <none>                   3m43s
    kube-proxy    1         1         1       1            1           kubernetes.io/os=linux   10m
    
    $ kubectl get po kata-deploy-5zwmq
    NAME                READY   STATUS    RESTARTS   AGE
    kata-deploy-5zwmq   1/1     Running   0          4m24
    
    kubectl logs kata-deploy-5zwmq
    copying kata artifacts onto host
    #!/bin/bash
    KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-fc.toml /opt/kata/bin/containerd-shim-kata-v2 "$@"
    #!/bin/bash
    KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-qemu.toml /opt/kata/bin/containerd-shim-kata-v2 "$@"
    #!/bin/bash
    KATA_CONF_FILE=/opt/kata/share/defaults/kata-containers/configuration-clh.toml /opt/kata/bin/containerd-shim-kata-v2 "$@"
    Add Kata Containers as a supported runtime for containerd
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
      runtime_type = "io.containerd.kata.v2"
      privileged_without_host_devices = true
      pod_annotations = ["io.katacontainers.*"]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata.options]
        ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration.toml"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-fc]
      runtime_type = "io.containerd.kata-fc.v2"
      privileged_without_host_devices = true
      pod_annotations = ["io.katacontainers.*"]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-fc.options]
        ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-fc.toml"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu]
      runtime_type = "io.containerd.kata-qemu.v2"
      privileged_without_host_devices = true
      pod_annotations = ["io.katacontainers.*"]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu.options]
        ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh]
      runtime_type = "io.containerd.kata-clh.v2"
      privileged_without_host_devices = true
      pod_annotations = ["io.katacontainers.*"]
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-clh.options]
        ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-clh.toml"
    node/alma8 labeled
    
    $ ll /opt/kata/bin/
    total 157532
    -rwxr-xr-x. 1 root root  4045032 Jul 19 06:10 cloud-hypervisor
    -rwxr-xr-x. 1 root root 42252997 Jul 19 06:12 containerd-shim-kata-v2
    -rwxr-xr-x. 1 root root  3290472 Jul 19 06:14 firecracker
    -rwxr-xr-x. 1 root root  2589888 Jul 19 06:14 jailer
    -rwxr-xr-x. 1 root root    16686 Jul 19 06:12 kata-collect-data.sh
    -rwxr-xr-x. 1 root root 37429099 Jul 19 06:12 kata-monitor
    -rwxr-xr-x. 1 root root 54149384 Jul 19 06:12 kata-runtime
    -rwxr-xr-x. 1 root root 17521656 Jul 19 06:18 qemu-system-x86_64
    

    Restart containerd to enable the new config:

    systemctl restart containerd
    

    Now I can start a Kata container from commadnline.

    sudo ctr image pull docker.io/library/hello-world:latest
    sudo ctr run --runtime io.containerd.run.kata-qemu.v2 -t --rm docker.io/library/hello-world:latest hello
    
    sudo ctr run --runtime io.containerd.run.kata-clh.v2 -t --rm docker.io/library/hello-world:latest hello
    
    ctr run --snapshotter devmapper --runtime io.containerd.run.kata-fc.v2 -t docker.io/library/hello-world:latest hello
    

    Start Deployment

    First I create a RuntimeClass for kata-fc then start a pod with this RuntimeClass.

    kubens default
    
    kubectl apply -f - <<EOF
    apiVersion: node.k8s.io/v1
    kind: RuntimeClass
    metadata:
      name: kata-fc
    handler: kata-fc
    EOF
    
    cat<<EOF | kubectl apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
      labels:
        app: untrusted
      name: www-kata-fc
    spec:
      runtimeClassName: kata-fc
      containers:
      - image: nginx:1.18
        name: www
        ports:
        - containerPort: 80
    EOF
    
    kubectl apply -f - <<EOF
    apiVersion: node.k8s.io/v1
    kind: RuntimeClass
    metadata:
      name: kata-qemu
    handler: kata-qemu
    EOF
    
    cat<<EOF | kubectl apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
      labels:
        app: untrusted
      name: www-kata-qemu
    spec:
      runtimeClassName: kata-qemu
      containers:
      - image: nginx:1.18
        name: www
        ports:
        - containerPort: 80
    EOF
    
    kubectl apply -f - <<EOF
    apiVersion: node.k8s.io/v1
    kind: RuntimeClass
    metadata:
      name: kata-clh
    handler: kata-clh
    EOF
    
    cat<<EOF | kubectl apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
      labels:
        app: untrusted
      name: www-kata-clh
    spec:
      runtimeClassName: kata-clh
      containers:
      - image: nginx:1.18
        name: www
        ports:
        - containerPort: 80
    EOF
    
    $ kubectl get po
    NAME            READY   STATUS    RESTARTS   AGE
    www-kata-clh    1/1     Running   0          59s
    www-kata-fc     1/1     Running   0          12s
    www-kata-qemu   1/1     Running   0          69s