How to backup and restore Prometheus?

Kubernetes
Page content

In this post I will show you how to take a backup from a running Prometheus and restore it.

I will assumes that you have a running Prometheus deployed with prometheus-operator in the monitoring namespace.

Enable Admin Api

First we need to enable the Prometheus’s admin api

kubectl -n monitoring patch prometheus prometheus-operator-prometheus \
  --type merge --patch '{"spec":{"enableAdminAPI":true}}'

In tmux or a separate window open a port forward to the admin api.

ubectl -n monitoring port-forward svc/prometheus-operator-prometheus 9090

Backup Prometheus data

Run following command to create a snapshot:

curl -XPOST http://localhost:9090/api/v2/admin/tsdb/snapshot
{"name":"20200731T123913Z-6e661e92759805f5"}

Find the snapshot and copy it to locally. The default folder is /prometheus/snapshots/ but you can find the data folder by finding the --storage.tsdb.path config in your deployment.

kubectl -n monitoring exec -it prometheus-prometheus-operator-prometheus-0 \
  -c prometheus -- /bin/sh -c \
  "ls /prometheus/snapshots/20200731T123913Z-6e661e92759805f5"
01EE25G1ZTKBFBBHFPHNBF99KJ  01EEFF7TE5ENDAGDR5K7ERW3BX
...

kubectl cp -n monitoring \
  prometheus-prometheus-operator-prometheus-0:/prometheus/snapshots/20200731T123913Z-6e661e92759805f5 \
  -c prometheus .

Restore Prometheus data

In a new Prometheus instance delete the data folder and copy the content of the snapshot:

kubectl -n newprom exec -it prometheus -- /bin/sh -c "rm -rf /prometheus/*"

kubectl -n newprom cp ./* prometheus:/prometheus/