Kubernetes Certificate Rotation
In this post I will show you how you can rotate your Kubernetes Engine Certificates.
Parst of the K8S Security series
- Part1: Best Practices to keeping Kubernetes Clusters Secure
- Part2: Kubernetes Hardening Guide with CIS 1.6 Benchmark
- Part3: RKE2 The Secure Kubernetes Engine
- Part4: RKE2 Install With cilium
- Part5: Kubernetes Certificate Rotation
- Part6: Hardening Kubernetes with seccomp
- Part7a: RKE2 Pod Security Policy
- Part7b: Kubernetes Pod Security Admission
- Part7c: Pod Security Standards using Kyverno
- Part8: Kubernetes Network Policy
- Part9: Kubernetes Cluster Policy with Kyverno
- Part10: Using Admission Controllers
- Part11a: Image security Admission Controller
- Part11b: Image security Admission Controller V2
- Part11c: Image security Admission Controller V3
- Part12: Continuous Image security
- Part13: K8S Logging And Monitoring
- Part14: Kubernetes audit logs and Falco
- Part15a Image Signature Verification with Connaisseur
- Part15b Image Signature Verification with Connaisseur 2.0
- Part15c Image Signature Verification with Kyverno
- Part16a Backup your Kubernetes Cluster
- Part16b How to Backup Kubernetes to git?
- Part17a Kubernetes and Vault integration
- Part17b Kubernetes External Vault integration
- Part18a: ArgoCD and kubeseal to encript secrets
- Part18b: Flux2 and kubeseal to encrypt secrets
- Part18c: Flux2 and Mozilla SOPS to encrypt secrets
- Part19: ArgoCD auto image updater
- Part20: Secure k3s with gVisor
- Part21: How to use imagePullSecrets cluster-wide??
- Part22: Automatically change registry in pod definition
By default, kubeadm generates all the certificates needed for a cluster to run. Client certificates generated by kubeadm expire after 1 year. The base concept is that you probably update for the next kubernetes version in a year.
Check certificate expiration
You can use the check-expiration
subcommand to check when certificates expire:
kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Apr 19, 2022 16:34 UTC 364d no
apiserver Apr 19, 2022 16:34 UTC 364d ca no
apiserver-etcd-client Apr 19, 2022 16:34 UTC 364d etcd-ca no
apiserver-kubelet-client Apr 19, 2022 16:34 UTC 364d ca no
controller-manager.conf Apr 19, 2022 16:34 UTC 364d no
etcd-healthcheck-client Apr 19, 2022 16:34 UTC 364d etcd-ca no
etcd-peer Apr 19, 2022 16:34 UTC 364d etcd-ca no
etcd-server Apr 19, 2022 16:34 UTC 364d etcd-ca no
front-proxy-client Apr 19, 2022 16:34 UTC 364d front-proxy-ca no
scheduler.conf Apr 19, 2022 16:34 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Apr 17, 2031 16:34 UTC 9y no
etcd-ca Apr 17, 2031 16:34 UTC 9y no
front-proxy-ca Apr 17, 2031 16:34 UTC 9y no
The kubernetes certificates are located under /etc/kubernetes/pki/
folder. You can check the certificates manulally:
find /etc/kubernetes/pki/ -type f -name "*.crt" -print |xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep Not'
The kubele certificate is not checkd by the abow command. It is located under the /var/lib/kubelet/pki/
folder.
Automatic certificate renewal
kubeadm
renews all the certificates during control plane upgrade. It is a best practice to upgrade your cluster frequently in order to stay secure. Kubernetes v1.8 and higher kubelet implements features for enabling rotation of its client and/or serving certificates.
kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap
kubectl create clusterrolebinding node-client-auto-approve-csr \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient \
--group=system:node-bootstrappers
kubectl create clusterrolebinding node-client-auto-renew-crt \
--clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient \
--group=system:nodes
nano /etc/systemd/system/kubelet.env
# or
nano /var/lib/kubelet/kubeadm-flags.env
...
KUBELET_EXTRA_ARGS=="--rotate-certificates=true --rotate-server-certificates=true"
systemctl restart kubelet
# OR
nano /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# OR
nano /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_EXTRA_ARGS=--rotate-certificates=true --rotate-server-certificates=true"
systemctl daemon-reload
systemctl restart kubelet
ps -ef | grep kubelet | grep "rotate-certificates"
root 14105 1 0 16:56 pts/0 00:00:00 bash -c while true ; do /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --pod-infra-container-image=k8s.gcr.io/pause:3.2 --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cgroup-driver=cgroupfs --fail-swap-on=false --resolv-conf=/etc/resolv.conf.override --rotate-certificates=true --rotate-server-certificates=true; sleep 5; done
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-pswns 27m kubernetes.io/kube-apiserver-client-kubelet system:node:node1 Approved,Issued
node-csr-cQYdjcH2F3kl-ysnzq2TlZOuUDCPgYU8cfKV1V0kqlE 47m kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:lzhuxv Approved,Issued
node-csr-f2J5HT9hg4CIKP0-0BtsEffBzg28VlUbesKJ4p_2mi0 47m kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:lzhuxv Approved,Issued
Manual certificate renewal
You can renew your certificates manually at any time with the kubeadm certs renew
command. This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki
If you are running an HA cluster, this command needs to be executed on all the control-plane nodes.
sudo kubeadm alpha certs renew all
RKE2 and K3S
By default, certificates in RKE2 and K3S expire in 12 months. If the certificates are expired or have fewer than 90 days remaining before they expire, the certificates are rotated when RKE2 is restarted. It is expected that you would be taking your hosts down periodically for patching and upgrading every few months. With regular updates the reboots should happen - but reality has shown that many of us do not patch / reboot for more than 3 months.. so the best practice is monitoring the certificate expiration.