kubernetes 1.24: Migrate from docker to cri-o

Kubernetes
Page content

With the new Kubernetes 1.24 and deprecation of dockershim, in this post I will show you how you can migrate your kubernetes cluster from docker to cri-o.

How to migrate

You have to be careful if you are on a single master node configuration. The cluster will be unavailable under the upgrade.

kubectl get nodes -o wide
NAME    STATUS   ROLES                  AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION                             CONTAINER-RUNTIME
k8s01   Ready    control-plane,master   78m     v1.20.4   10.65.79.164   <none>        CentOS Linux 8       4.18.0-240.15.1.el8_3.centos.plus.x86_64   docker://20.10.5
k8s02   Ready    control-plane,master   64m     v1.20.4   10.65.79.131   <none>        CentOS Linux 8       4.18.0-240.15.1.el8_3.centos.plus.x86_64   docker://20.10.5
k8s03   Ready    control-plane,master   4m16s   v1.20.4   10.65.79.244   <none>        CentOS Linux 8       4.18.0-240.15.1.el8_3.centos.plus.x86_64   docker://20.10.5

First we will cordon and drain the node:

kubectl cordon k8s01
kubectl drain k8s01 --ignore-daemonsets

kubectl get nodes
NAME    STATUS                      ROLES                  AGE    VERSION
k8s01   Ready,SchedulingDisabled    control-plane,master   83m    v1.20.4
k8s02   Ready                       control-plane,master   69m    v1.20.4
k8s03   Ready                       control-plane,master   9m30s  v1.20.4

Stop the kubelet sevice and remove docker:

sudo systemctl stop kubelet
sudo systemctl status kubelet

apt purge docker-ce docker-ce-cli
OR
yum remove docker-ce docker-ce-cli

Install and configure cri-o:

## Install cri-o to centos/alma linux/rocky linux
VERSION=1.24
OS=CentOS_7
# OR
OS=CentOS_8
sudo curl -L -o /etc/yum.repos.d/devel_kubic_libcontainers_stable.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/$OS/devel:kubic:libcontainers:stable.repo
sudo curl -L -o /etc/yum.repos.d/devel_kubic_libcontainers_stable_cri-o_${VERSION}.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:${VERSION}/$OS/devel:kubic:libcontainers:stable:cri-o:${VERSION}.repo

yum install cri-o

## Install cri-o to ubuntu/debian
VERSION=1.24
OS=Debian_Unstable
# OR
OS=xUbuntu_20.04
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | apt-key add -
curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | apt-key add -

apt-get update
apt-get install cri-o cri-o-runc

You nee the same cgroup manager in cri-o and kubeadm. The default for kubeadm is cgroupfs and for cri-o the default is systemd. In this example I configured cri-o for cgroupfs.

nano /etc/crio/crio.conf
[crio.runtime]
conmon_cgroup = "pod"
cgroup_manager = "cgroupfs"

nano /etc/containers/registries.conf
registries = [
  "quay.io",
  "docker.io"
]

If you want to use systemd:

echo "KUBELET_EXTRA_ARGS=--cgroup-driver=systemd" | tee /etc/sysconfig/kubelet

Prepare the system for cri-o:

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system

echo "runtime-endpoint: unix:///var/run/crio/crio.sock" > /etc/crictl.yaml
crictl ps

# Start cri-o
systemctl enable --now crio
systemctl status crio

Change runtime in kubeadm config:

nano /etc/sysconfig/kubelet
# add the following flags to KUBELET_KUBEADM_ARGS variable
KUBELET_KUBEADM_ARGS="... --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --runtime-request-timeout=5m --cgroup-driver=systemd "

Start kubelet:

sudo systemctl start kubelet

Check if the new runtime on the node:

kubectl describe node k8s01

Uncordon the node to mark it schedulable agen:

kubectl uncordon k8s01

Once you changed the runetime on all the nodes you are done.