Bulletproof Kubernetes Backups: K8up, Self-Hosted REST Server, and Full Prometheus Observability

Kubernetes
6 min read
Bulletproof Kubernetes Backups: K8up, Self-Hosted REST Server, and Full Prometheus Observability
Page content

Managing persistent storage backups in a dynamic cloud-native environment can scale out of control rapidly. While tools like Restic provide incredible speed, encryption, and deduplication, orchestrating backups container by container with custom shell scripts is fragile.

K8up elegantly solves this by transforming Restic into a native Kubernetes automation system. In this comprehensive guide, we will break down how K8up works, deploy a self-hosted Restic REST backend with a production-grade Web UI via Docker, and configure enterprise monitoring using Prometheus, Grafana, and Alertmanager.

1. What is K8up?

K8up is an open-source, CNCF Sandbox Kubernetes backup operator built entirely on top of Restic. It replaces localized snapshot infrastructure with Kubernetes Custom Resource Definitions (CRDs).

Instead of configuring isolated jobs, developers submit YAML declarations for operations like Schedule, Backup, Check, and Restore. K8up intercepts these manifests, monitors the relevant namespaces, provisions temporary worker pods to mount targeting PersistentVolumeClaims (PVCs) as read-only, and directly pipelines data to your remote destination.

2. Supported Storage Backends

Because K8up relies on Restic under the hood, it shares Restic’s universal backend engine. Supported options include:

  • S3-Compatible Object Storage: AWS S3, MinIO, Wasabi, Google Cloud Storage, or Backblaze B2.
  • Network Encrypted Storage: SFTP (Secure File Transfer Protocol).
  • The Restic REST Protocol: A specialized, high-velocity append-only HTTP protocol engineered explicitly for processing high-frequency Restic client streams.

We will use the REST protocol because it offers extreme performance advantages over SFTP and is vastly simpler to host than an enterprise object storage array.

3. Deploy a simple app

We’re going to deploy a sample application written in Go using the deployment.yaml file shown below. This deployment doesn’t need neither a Service nor an Ingress:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fortune-go
spec:
  template:
    spec:
      containers:
      - image: registry.gitlab.com/vshn/applications/fortune-go:latest
        imagePullPolicy: Always
        name: fortune-container
        ports:
        - containerPort: 8080
    metadata:
      labels:
        app: fortune-go
      annotations:
        k8up.io/backupcommand: fortune
        k8up.io/file-extension: .txt
  selector:
    matchLabels:
      app: fortune-go
  strategy:
    type: Recreate

Apply the deployment:

kubectl apply -f deployment.yaml

4. Deploying the Restic REST Server via Docker

Docker provides the absolute fastest implementation path for setting up a dedicated remote repository target.

The Docker Compose Core

Create a local working directory and drop the following code block inside a file named docker-compose.yml. This exposes the core REST service and maps local server paths for cold storage.

services:
  rest-server:
    image: restic/rest-server:latest
    container_name: restic-backend
    restart: unless-stopped
    ports:
      - "8000:8000"
    volumes:
      - ./backups:/data
    environment:
      # Isolation configuration: Ensures users can only access their specific explicitly assigned directories
      - OPTIONS=--private-repos --prometheus

Forcing User Authentication via htpasswd

The core image intercepts authentication requests natively by verifying incoming Basic Auth details against a localized .htpasswd asset. Run these quick terminal entries before triggering the stack:

# Provision storage layout
mkdir -p ./backups

# Pull a temporary htpasswd generator to cleanly hash security strings
docker run --rm -ti trantor/htpasswd -B -c ./backups/.htpasswd cluster-backup-user

(Provide a strong, robust password when prompted to anchor this user account securely).

Launch the backend instance immediately:

docker compose up -d rest-server

5. Deploying a Web UI with Built-In Authentication

The base REST server has no graphical state engine. To visualize repositories, audit snapshots, and observe raw disk footprint metrics visually, we will link it to Backrest—a secure, multi-repo Web UI wrapper natively featuring built-in application-layer user authentication.

Append the Backrest service directly to your existing docker-compose.yml payload:

  backrest-ui:
    image: ghcr.io/garethgeorge/backrest:latest
    container_name: restic-webui
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - ./backups:/data
      - ./backrest-config:/config
    environment:
      # Declares rigid instance credentials required for first-load browser access
      - BACKREST_AUTH_USER=admin
      - BACKREST_AUTH_PASS=YourComplexVisualPasswordHere!

Reload your container configurations:

docker compose up -d

You can now immediately route your desktop browser to http://<your-server-ip>:8080 to interact with your data.

6. Installing and Configuring K8up

Now that your visual target storage engine is operational, install the K8up engine on your target cluster.

Helm Deployment

Map the official project charts and install the operator into an isolated administrative namespace:

helm repo add k8up-io https://k8up.iohelm-charts
helm repo update

helm install k8up k8up-io/k8up \
  --namespace k8up-system \
  --create-namespace \
  --set metrics.serviceMonitor.enabled=true # Automatically exposes endpoints to Prometheus Operator

Storage Vault Mapping

Create a secure credentials profile inside the target workspace containing the microservices you intend to protect (e.g., production-services):

apiVersion: v1
kind: Secret
metadata:
  name: backup-vault-auth
  namespace: production-services
type: Opaque
stringData:
  # The explicit symmetric key utilized to perform client-side encryption on your blocks
  repo-password: SuperStrongEncryptionKey321!
  # Basic Authentication configurations matching your Docker htpasswd container
  username: cluster-backup-user
  password: DockerHtpasswdPasswordCreatedEarlier

Automation Schedule

Apply a standard Schedule manifest to run automated snapshots every night at 3:00 AM:

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: fortune-go-application-backup
  namespace: production-services # Ensure this matches your deployment's namespace
spec:
  backend:
    rest:
      # Reuses the Docker rest-server credentials established earlier
      url: http://cluster-backup-user:DockerHtpasswdPasswordCreatedEarlier@your-server-ip:8000/k8s-production-repo
    repoPasswordSecretRef:
      name: backup-vault-auth
      key: repo-password
  backup:
    # Triggers the backup every day at 3:15 AM
    schedule: "15 3 * * *" 
    failedJobsHistoryLimit: 3
    successfulJobsHistoryLimit: 3

kubectl apply -f schedule.yaml binds this configuration. K8up will handle volume parsing completely automatically moving forward.

7. Configuring Prometheus Monitoring & Grafana

Unmonitored backups are effectively non-existent. K8up exposes real-time Prometheus time-series metrics (k8up_jobs_total, k8up_backups_total).

If your monitoring solution utilizes the standard Kube-Prometheus-Stack pattern, configure a PodMonitor object inside your operator namespace to capture operational states:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: k8up-operator-monitor
  namespace: k8up-system
  labels:
    release: prometheus # Changes based on your specific helm release name
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: k8up
  podMetricsEndpoints:
  - port: metrics
    interval: 30s

Importing the Grafana Dashboard

To observe success ratios and compression statistics without writing raw queries, import the pre-configured Official K8up Dashboard from Grafana Labs:

  1. Log into your active Grafana panel dashboard interface.
  2. Choose Dashboards -> New -> Import.
  3. Provide the official Community Dashboard ID: 24597 (or 20166 depending on older schema variations).
  4. Map the dropdown connection to your underlying Prometheus data source and click Import.

8. Creating Alerting Rules for Failed Backups

To ensure immediate awareness if an incremental backup script terminates with error states, apply a native PrometheusRule layout. This configuration hooks straight into Alertmanager to route alerts to targets like Slack, PagerDuty, or Webhooks:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: k8up-backup-alerts
  namespace: k8up-system
  labels:
    role: alert-rules
    release: prometheus
spec:
  groups:
  - name: k8up.rules
    rules:
    - alert: K8upBackupJobFailed
      expr: k8up_jobs_total{condition="failed", type="backup"} > 0
      for: 2m
      labels:
        severity: critical
        tier: platform
      annotations:
        summary: "Kubernetes backup job failed in namespace {{ \$labels.namespace }}"
        description: "A scheduled K8up Restic execution failed to process completely. Inspect the operator logs immediately via 'kubectl logs -n k8up-system' to prevent loss of target application context."

Apply via kubectl apply -f alerts.yaml. Your backup topology is now fully automated, visual, secure, and observable at scale.