Kubernetes Certificate Rotation

Page content

In this post I will show you how you can rotate your Kubernetes Engine Certificates.

Parts of the K8S Security Lab series

Container Runetime Security
Advanced Kernel Security
Network Security
Secure Kubernetes Install
User Security
Image Security
  • Part1: Image security Admission Controller
  • Part2: Image security Admission Controller V2
  • Part3: Image security Admission Controller V3
  • Part4: Continuous Image security
  • Part5: trivy-operator 1.0
  • Part6: trivy-operator 2.1: Trivy-operator is now an Admisssion controller too!!!
  • Part7: trivy-operator 2.2: Patch release for Admisssion controller
  • Part8: trivy-operator 2.3: Patch release for Admisssion controller
  • Part8: trivy-operator 2.4: Patch release for Admisssion controller
  • Part8: trivy-operator 2.5: Patch release for Admisssion controller
  • Part9_ Image Signature Verification with Connaisseur
  • Part10: Image Signature Verification with Connaisseur 2.0
  • Part11: Image Signature Verification with Kyverno
  • Part12: How to use imagePullSecrets cluster-wide??
  • Part13: Automatically change registry in pod definition
  • Part14: ArgoCD auto image updater
    Pod Security
    Secret Security
    Monitoring and Observability
    Backup

    By default, kubeadm generates all the certificates needed for a cluster to run. Client certificates generated by kubeadm expire after 1 year. The base concept is that you probably update for the next kubernetes version in a year.

    Check certificate expiration

    You can use the check-expiration subcommand to check when certificates expire:

    kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Apr 19, 2022 16:34 UTC   364d                                    no
apiserver                  Apr 19, 2022 16:34 UTC   364d            ca                      no
apiserver-etcd-client      Apr 19, 2022 16:34 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Apr 19, 2022 16:34 UTC   364d            ca                      no
controller-manager.conf    Apr 19, 2022 16:34 UTC   364d                                    no
etcd-healthcheck-client    Apr 19, 2022 16:34 UTC   364d            etcd-ca                 no
etcd-peer                  Apr 19, 2022 16:34 UTC   364d            etcd-ca                 no
etcd-server                Apr 19, 2022 16:34 UTC   364d            etcd-ca                 no
front-proxy-client         Apr 19, 2022 16:34 UTC   364d            front-proxy-ca          no
scheduler.conf             Apr 19, 2022 16:34 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Apr 17, 2031 16:34 UTC   9y              no
etcd-ca                 Apr 17, 2031 16:34 UTC   9y              no
front-proxy-ca          Apr 17, 2031 16:34 UTC   9y              no

    The kubernetes certificates are located under /etc/kubernetes/pki/ folder. You can check the certificates manulally:

    find /etc/kubernetes/pki/ -type f -name "*.crt" -print |xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep Not'

    The kubele certificate is not checkd by the abow command. It is located under the /var/lib/kubelet/pki/ folder.

    Automatic certificate renewal

    kubeadm renews all the certificates during control plane upgrade. It is a best practice to upgrade your cluster frequently in order to stay secure. Kubernetes v1.8 and higher kubelet implements features for enabling rotation of its client and/or serving certificates.

    kubectl create clusterrolebinding kubelet-bootstrap \
    --clusterrole=system:node-bootstrapper \
    --user=kubelet-bootstrap

kubectl create clusterrolebinding node-client-auto-approve-csr \
    --clusterrole=system:certificates.k8s.io:certificatesigningrequests:nodeclient \
    --group=system:node-bootstrappers

kubectl create clusterrolebinding node-client-auto-renew-crt \
    --clusterrole=system:certificates.k8s.io:certificatesigningrequests:selfnodeclient \
    --group=system:nodes

    nano /etc/systemd/system/kubelet.env
# or
nano /var/lib/kubelet/kubeadm-flags.env
...
KUBELET_EXTRA_ARGS=="--rotate-certificates=true --rotate-server-certificates=true"

systemctl restart kubelet

# OR
nano /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
# OR
nano /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_EXTRA_ARGS=--rotate-certificates=true --rotate-server-certificates=true"

systemctl daemon-reload
systemctl restart kubelet

ps -ef | grep kubelet | grep "rotate-certificates"
root      14105      1  0 16:56 pts/0    00:00:00 bash -c while true ; do /usr/bin/kubelet  --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --pod-infra-container-image=k8s.gcr.io/pause:3.2 --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cgroup-driver=cgroupfs --fail-swap-on=false --resolv-conf=/etc/resolv.conf.override --rotate-certificates=true --rotate-server-certificates=true; sleep 5; done

    kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR                 CONDITION
csr-pswns                                              27m   kubernetes.io/kube-apiserver-client-kubelet   system:node:node1         Approved,Issued
node-csr-cQYdjcH2F3kl-ysnzq2TlZOuUDCPgYU8cfKV1V0kqlE   47m   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:lzhuxv   Approved,Issued
node-csr-f2J5HT9hg4CIKP0-0BtsEffBzg28VlUbesKJ4p_2mi0   47m   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:lzhuxv   Approved,Issued

    Manual certificate renewal

    You can renew your certificates manually at any time with the kubeadm certs renew command. This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki If you are running an HA cluster, this command needs to be executed on all the control-plane nodes.

    sudo kubeadm alpha certs renew all

    RKE2 and K3S

    By default, certificates in RKE2 and K3S expire in 12 months. If the certificates are expired or have fewer than 90 days remaining before they expire, the certificates are rotated when RKE2 is restarted. It is expected that you would be taking your hosts down periodically for patching and upgrading every few months. With regular updates the reboots should happen - but reality has shown that many of us do not patch / reboot for more than 3 months.. so the best practice is monitoring the certificate expiration.