How to Backup Kubernetes to git?

In this tutorial I will show you how you can backup the kubernetes object to git as yaml-s.

Parts of the K8S Security Lab series

Container Runetime Security

• Part1: How to deploy CRI-O with Firecracker?
• Part2: How to deploy CRI-O with gVisor?
• Part3: How to deploy containerd with Firecracker?
• Part4: How to deploy containerd with gVisor?
• Part5: How to deploy containerd with kata containers?

Advanced Kernel Security

• Part1: Hardening Kubernetes with seccomp
• Part2: Linux user namespace management wit CRI-O in Kubernetes
• Part3: Hardening Kubernetes with seccomp

Network Security

• Part1: RKE2 Install With Calico
• Part2: RKE2 Install With Cilium
• Part3: CNI-Genie: network separation with multiple CNI
• Part3: Configurre network wit nmstate operator
• Part3: Kubernetes Network Policy
• Part4: Kubernetes with external Ingress Controller with vxlan
• Part4: Kubernetes with external Ingress Controller with bgp
• Part4: Central authentication with oauth2-proxy
• Part5: Secure your applications with Pomerium Ingress Controller
• Part6: CrowdSec Intrusion Detection System (IDS) for Kubernetes
• Part7: Kubernetes audit logs and Falco

Secure Kubernetes Install

• Part1: Best Practices to keeping Kubernetes Clusters Secure
• Part2: Kubernetes Secure Install
• Part3: Kubernetes Hardening Guide with CIS 1.6 Benchmark
• Part4: Kubernetes Certificate Rotation

User Security

• Part1: How to create kubeconfig?
• Part2: How to create Users in Kubernetes the right way?
• Part3: Kubernetes Single Sign-on with Pinniped OpenID Connect
• Part4: Kubectl authentication with Kuberos Depricated !!
• Part5: Kubernetes authentication with Keycloak and gangway Depricated !!
• Part6: kube-openid-connect 1.0 Depricated !!

Image Security

Pod Security

• Part1: Using Admission Controllers
• Part2: RKE2 Pod Security Policy
• Part3: Kubernetes Pod Security Admission
• Part4: Kubernetes: How to migrate Pod Security Policy to Pod Security Admission?
• Part5: Pod Security Standards using Kyverno
• Part6: Kubernetes Cluster Policy with Kyverno

Secret Security

• Part1: Kubernetes and Vault integration
• Part2: Kubernetes External Vault integration
• Part3: ArgoCD and kubeseal to encript secrets
• Part4: Flux2 and kubeseal to encrypt secrets
• Part5: Flux2 and Mozilla SOPS to encrypt secrets

Monitoring and Observability

• Part6: K8S Logging And Monitoring
• Part7: Install Grafana Loki with Helm3

Backup

• Part1: Backup your Kubernetes Cluster
• Part2: How to Backup Kubernetes to git?

Thanky to Maxim Levchenko ther is a grate tool called kube-dump that is dump all of the kubernetes objects to a git repository as yaml. We will use this tool to backup.

Key features:

• Saving is done only for those resources to which you have read access.
• You can pass a list of namespaces as an input, otherwise all available for your context will be used.
• Both namespace resources and global cluster resources are subject to persistence.
• You can use the utility locally as a regular script or run it in a container or in a kubernetes cluster, for example, as a CronJob.
• It can create archives and rotate them after itself.
• Can commit state to git repository and push to remote repository.
• You can specify a specific list of cluster resources for unloading.

kubectl create ns kube-dump
kubectl -n kube-dump apply -f \
  https://raw.githubusercontent.com/WoozyMasta/kube-dump/master/deploy/cluster-role-view.yaml

Deploy with git repository oauth token

Project access tokens are supported for self-managed instances on Free and above. They are also supported on GitLab SaaS Premium and above. If you use GitLab SaaS on Free you can us Personal access token instead of Project Access Token.

As an example, I will use authorization in GitLab using the Project Access Token, so we will create a secret with the repository address and an authorization token:

kubectl -n kube-dump create secret generic kube-dump \
  --from-literal=GIT_REMOTE_URL=https://oauth2:$TOKEN@corp-gitlab.com/devops/cluster-01.git

Before Kubernetes 1.22 CronJob’s timezone is always UTC. If you want to change this use cronjobber Since Kubernetes 1.22 you can add timezon in cronjob with CRON_TZ variable.

Let’s set up a CronJob in which we indicate the frequency of the task launch:

wget https://github.com/WoozyMasta/kube-dump/blob/master/deploy/cronjob-git-token.yaml

nano cronjob-git-token.yaml
...
spec:
  schedule: "0 1 * * *"

kubectl apply -f cronjob-git-token.yaml -n kube-dump

Deploy with git repository write allowed ssh key

Generate ssh key:

mkdir -p ./.ssh
chmod 0700 ./.ssh
ssh-keygen -t ed25519 -C "kube-dump" -f ./.ssh/kube-dump
cat ./.ssh/kube-dump.pub

kubectl -n kube-dump create secret generic kube-dump-key \
  --from-file=./.ssh/kube-dump \
  --from-file=./.ssh/kube-dump.pub

Create pvc for store data such as cache:

kubectl apply -n kube-dump -f deploy/pvc.yaml

And apply the cron job manifest, previously you could set up environment variables:

wget https://github.com/WoozyMasta/kube-dump/blob/master/deploy/cronjob-git-key.yaml

nano cronjob-git-key.yaml
...
spec:
  schedule: "0 1 * * *"
...
              env:
                - name: MODE
                  value: "dump"
                - name: DESTINATION_DIR
                  value: "/data/dump"
                - name: GIT_PUSH
                  value: "true"
                - name: GIT_BRANCH
                  value: "master"
                - name: GIT_COMMIT_USER
                  value: "Kube Dump"
                - name: GIT_COMMIT_EMAIL
                  value: "kube@dump.local"
                - name: GIT_REMOTE_URL
                  value: "git@corp-gitlab.com:devops/cluster-bkp.git"

kubectl apply -f cronjob-git-key.yaml -n kube-dump