How to Backup Kubernetes to git?
In this tutorial I will show you how you can backup the kubernetes object to git as yaml-s.
Parst of the K8S Security series
- Part1: Best Practices to keeping Kubernetes Clusters Secure
- Part2: Kubernetes Hardening Guide with CIS 1.6 Benchmark
- Part3: RKE2 The Secure Kubernetes Engine
- Part4: RKE2 Install With cilium
- Part5: Kubernetes Certificate Rotation
- Part6: Hardening Kubernetes with seccomp
- Part7a: RKE2 Pod Security Policy
- Part7b: Kubernetes Pod Security Admission
- Part7c: Pod Security Standards using Kyverno
- Part8: Kubernetes Network Policy
- Part9: Kubernetes Cluster Policy with Kyverno
- Part10: Using Admission Controllers
- Part11a: Image security Admission Controller
- Part11b: Image security Admission Controller V2
- Part11c: Image security Admission Controller V3
- Part12: Continuous Image security
- Part13: K8S Logging And Monitoring
- Part14: Kubernetes audit logs and Falco
- Part15a Image Signature Verification with Connaisseur
- Part15b Image Signature Verification with Connaisseur 2.0
- Part15c Image Signature Verification with Kyverno
- Part16a Backup your Kubernetes Cluster
- Part16b How to Backup Kubernetes to git?
- Part17a Kubernetes and Vault integration
- Part17b Kubernetes External Vault integration
- Part18a: ArgoCD and kubeseal to encript secrets
- Part18b: Flux2 and kubeseal to encrypt secrets
- Part18c: Flux2 and Mozilla SOPS to encrypt secrets
- Part19: ArgoCD auto image updater
- Part20: Secure k3s with gVisor
- Part21: How to use imagePullSecrets cluster-wide??
- Part22: Automatically change registry in pod definition
Thanky to Maxim Levchenko ther is a grate tool called kube-dump that is dump all of the kubernetes objects to a git repository as yaml. We will use this tool to backup.
Key features:
- Saving is done only for those resources to which you have read access.
- You can pass a list of namespaces as an input, otherwise all available for your context will be used.
- Both namespace resources and global cluster resources are subject to persistence.
- You can use the utility locally as a regular script or run it in a container or in a kubernetes cluster, for example, as a CronJob.
- It can create archives and rotate them after itself.
- Can commit state to git repository and push to remote repository.
- You can specify a specific list of cluster resources for unloading.
kubectl create ns kube-dump
kubectl -n kube-dump apply -f \
https://raw.githubusercontent.com/WoozyMasta/kube-dump/master/deploy/cluster-role-view.yaml
Deploy with git repository oauth token
Project access tokens are supported for self-managed instances on Free and above. They are also supported on GitLab SaaS Premium and above. If you use GitLab SaaS on Free you can us Personal access token instead of Project Access Token.
As an example, I will use authorization in GitLab using the Project Access Token, so we will create a secret with the repository address and an authorization token:
kubectl -n kube-dump create secret generic kube-dump \
--from-literal=GIT_REMOTE_URL=https://oauth2:$TOKEN@corp-gitlab.com/devops/cluster-01.git
Before Kubernetes 1.22 CronJob’s timezone is always UTC. If you want to change this use cronjobber Since Kubernetes 1.22 you can add timezon in cronjob with
CRON_TZ
variable.
Let’s set up a CronJob in which we indicate the frequency of the task launch:
wget https://github.com/WoozyMasta/kube-dump/blob/master/deploy/cronjob-git-token.yaml
nano cronjob-git-token.yaml
...
spec:
schedule: "0 1 * * *"
kubectl apply -f cronjob-git-token.yaml -n kube-dump
Deploy with git repository write allowed ssh key
Generate ssh key:
mkdir -p ./.ssh
chmod 0700 ./.ssh
ssh-keygen -t ed25519 -C "kube-dump" -f ./.ssh/kube-dump
cat ./.ssh/kube-dump.pub
kubectl -n kube-dump create secret generic kube-dump-key \
--from-file=./.ssh/kube-dump \
--from-file=./.ssh/kube-dump.pub
Create pvc for store data such as cache:
kubectl apply -n kube-dump -f deploy/pvc.yaml
And apply the cron job manifest, previously you could set up environment variables:
wget https://github.com/WoozyMasta/kube-dump/blob/master/deploy/cronjob-git-key.yaml
nano cronjob-git-key.yaml
...
spec:
schedule: "0 1 * * *"
...
env:
- name: MODE
value: "dump"
- name: DESTINATION_DIR
value: "/data/dump"
- name: GIT_PUSH
value: "true"
- name: GIT_BRANCH
value: "master"
- name: GIT_COMMIT_USER
value: "Kube Dump"
- name: GIT_COMMIT_EMAIL
value: "kube@dump.local"
- name: GIT_REMOTE_URL
value: "git@corp-gitlab.com:devops/cluster-bkp.git"
kubectl apply -f cronjob-git-key.yaml -n kube-dump