Migrating on-prem kubernetes from Docker to Containerd

Written by

Written by Dan Jones

‚Äč

			Migrating on-prem kubernetes from Docker to Containerd
docker to containerd

Published on our Cloud Native Blog.
Tagged with

Kubernetes 1.22 was recently released and the steady march towards deprecating the docker shim progresses. On-premise cluster admins may be a bit daunted by the task of migrating to a new container runtime, but fear not! We recently helped a client through the process and wanted to share our findings to help make the migration easier for everyone.

docker to containerd

Preparations

Before you start re-adding nodes to the cluster, it’d be wise to set the kubelet-config configmap to use the systemd cgroup driver. You can find instructions on that here:

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#migrating-to-the-systemd-driver

Migration

Start by picking your least favourite worker node, cordoning it and waiting for the workloads to drain from it.

kubectl drain <node-name> --ignore-daemonsets

Then remove it from the cluster.

kubectl delete node <node-name>

Now reset the removed node and clean it up:

kubeadm reset

Stop docker and kubelet

systemctl stop kubelet
systemctl disable --now docker

Remove docker and reinstall containerd. Unless you mark the containerd package as manually installed, some package managers remove it as a dependency of docker-ce.

Make sure you clean up the data that isn’t cleared when running reset

  • /etc/cni/net.d/*
  • /opt/cni
  • /var/lib/docker
  • /var/lib/dockershim

Prepare the system for containerd as described here: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

Now we generate the default config for containerd

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

edit the containerd config file to enable the systemd cgroup driver for the CRI

In /etc/containerd/config.toml

And below the line:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]

Add:

SystemdCgroup = true

Start and enable containerd:

systemctl enable --now containerd

Once containerd is up and running you’ll need to re-install kubelet and kubeadm. I found that kubelet only reliably detected the new container runtime when reinstalled. If anyone knows of a way to get this to work without reinstalling, send your answers on a postcard please :)

Finally rejoin the node to the cluster. You should be able to verify the Node’s CRI with

kubectl get nodes -o wide

Once this is all done for your worker nodes, your control plane nodes are next. The steps are mostly the same, but if you run etcd in-cluster, it may be a good idea to reset your nodes before you delete them, to ensure they safely exit the etcd cluster.

Now with Docker removed and your cluster fully migrated to Containerd, you have much less to worry about in the upcoming kubernetes updates! You still have a little bit of time of course, with the earliest possible deprecation being later this year with kubernetes version 1.23.

Get started with Jetstack

Enquire about Subscription

Contact us