Kubernetes and Vault integration
In this post I will show you how you can integrate HashiCorp Vault to Kubernetes easily thanks to Bank-Vaults made by Banzaicloud.
Parts of the K8S Security Lab series
Container Runetime Security
- Part1: How to deploy CRI-O with Firecracker?
- Part2: How to deploy CRI-O with gVisor?
- Part3: How to deploy containerd with Firecracker?
- Part4: How to deploy containerd with gVisor?
- Part5: How to deploy containerd with kata containers?
Advanced Kernel Security
- Part1: Hardening Kubernetes with seccomp
- Part2: Linux user namespace management wit CRI-O in Kubernetes
- Part3: Hardening Kubernetes with seccomp
Network Security
- Part1: RKE2 Install With Calico
- Part2: RKE2 Install With Cilium
- Part3: CNI-Genie: network separation with multiple CNI
- Part3: Configurre network wit nmstate operator
- Part3: Kubernetes Network Policy
- Part4: Kubernetes with external Ingress Controller with vxlan
- Part4: Kubernetes with external Ingress Controller with bgp
- Part4: Central authentication with oauth2-proxy
- Part5: Secure your applications with Pomerium Ingress Controller
- Part6: CrowdSec Intrusion Detection System (IDS) for Kubernetes
- Part7: Kubernetes audit logs and Falco
Secure Kubernetes Install
- Part1: Best Practices to keeping Kubernetes Clusters Secure
- Part2: Kubernetes Secure Install
- Part3: Kubernetes Hardening Guide with CIS 1.6 Benchmark
- Part4: Kubernetes Certificate Rotation
User Security
- Part1: How to create kubeconfig?
- Part2: How to create Users in Kubernetes the right way?
- Part3: Kubernetes Single Sign-on with Pinniped OpenID Connect
- Part4: Kubectl authentication with Kuberos Depricated !!
- Part5: Kubernetes authentication with Keycloak and gangway Depricated !!
- Part6: kube-openid-connect 1.0 Depricated !!
Image Security
Pod Security
- Part1: Using Admission Controllers
- Part2: RKE2 Pod Security Policy
- Part3: Kubernetes Pod Security Admission
- Part4: Kubernetes: How to migrate Pod Security Policy to Pod Security Admission?
- Part5: Pod Security Standards using Kyverno
- Part6: Kubernetes Cluster Policy with Kyverno
Secret Security
- Part1: Kubernetes and Vault integration
- Part2: Kubernetes External Vault integration
- Part3: ArgoCD and kubeseal to encript secrets
- Part4: Flux2 and kubeseal to encrypt secrets
- Part5: Flux2 and Mozilla SOPS to encrypt secrets
Monitoring and Observability
- Part6: K8S Logging And Monitoring
- Part7: Install Grafana Loki with Helm3
Backup
In a previous post I talked about how Kubernetes cluster store the Kubernetes Secrets in the etcd as base64 encoded text and not encrypted. This is the reason why using an external secret store should be a good idea.
What is Bank-Vaults
Bank-Vaults provides various tools for Hashicorp Vault to make its use easier. It is a wrapper for the official Vault client with automatic token renewal, built in Kubernetes support, and a dynamic database credential provider.
Vhat is Hashicorp Vault
HashiCorp Vault is a secrets management solution that brokers access for both humans and machines, through programmatic access, to systems. Secrets can be stored, dynamically generated, and in the case of encryption, keys can be consumed as a service without the need to expose the underlying key materials.
Install Bank-Vaults Operator
Ther is a Kubernetes Helm chart to deploy the Banzai Cloud Vault Operator. We will use this for deploy the HashiCorp Vault in HA mode with etcd as storage backend. As a dependency the chart installs an etcd operator that runs as root so we need to use my predifinde PSP to allow this.
---
apiVersion: v1
kind: Namespace
metadata:
name: vault
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: psp-rolebinding-vault
namespace: vault
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system-unrestricted-psp-role
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:serviceaccounts
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: vault-operator
namespace: vault
spec:
repo: "https://kubernetes-charts.banzaicloud.com"
chart: vault-operator
targetNamespace: vault
valuesContent: |-
etcd-operator:
enabled: "true"
etcdOperator:
commandArgs:
cluster-wide: "true"
psp:
enabled: true
vaultSA: "vault"
When the operator runs correctly we can deploy the CRD to create teh Vault cluster
apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
name: "vault"
spec:
size: 2
image: vault:1.6.2
# Specify the ServiceAccount where the Vault Pod and the Bank-Vaults configurer/unsealer is running
serviceAccount: vault
# Specify how many nodes you would like to have in your etcd cluster
# NOTE: -1 disables automatic etcd provisioning
etcdSize: 1
#resources:
# vault:
# requests:
# memory: "256Mi"
# cpu: "100m"
# limits:
# memory: "512Mi"
# cpu: "250m"
etcdPVCSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
etcdAnnotations:
etcd.database.coreos.com/scope: clusterwide
etcdVersion: "3.3.17"
# Support for distributing the generated CA certificate Secret to other namespaces.
# Define a list of namespaces or use ["*"] for all namespaces.
caNamespaces:
- "demo-app"
- "default"
# Describe where you would like to store the Vault unseal keys and root token.
unsealConfig:
kubernetes:
secretNamespace: vault
# A YAML representation of a final vault config file.
# See https://www.vaultproject.io/docs/configuration/ for more information.
config:
storage:
etcd:
address: https://etcd-cluster:2379
ha_enabled: "true"
etcd_api: "v3"
listener:
tcp:
address: "0.0.0.0:8200"
tls_cert_file: /vault/tls/server.crt
tls_key_file: /vault/tls/server.key
api_addr: https://vault.vault:8200
telemetry:
statsd_address: localhost:9125
ui: true
externalConfig:
policies:
- name: allow_secrets
rules: path "secret/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
# The auth block allows configuring Auth Methods in Vault.
# See https://www.vaultproject.io/docs/auth/index.html for more information.
auth:
- type: kubernetes
roles:
# Allow every pod in the default namespace to use the secret kv store
- name: default
bound_service_account_names: default
bound_service_account_namespaces: "*"
policies: allow_secrets
ttl: 1h
secrets:
- path: secret
type: kv
description: General secrets
options:
version: 2
Deploy the mutating webhook
Banzaicloud created a mutating webhook to automate the injection of the secrets from Vault.
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: vault-secrets-webhook
namespace: vault
spec:
repo: "https://kubernetes-charts.banzaicloud.com"
chart: vault-secrets-webhook
targetNamespace: vault
Install vault cli and create secret
sudo yum install -y yum-utils
# OR
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
yum install -y vault
Configure the client to connect to the server:
export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
export VAULT_CACERT=$PWD/vault-ca.crt
export VAULT_ADDR=https://127.0.0.1:8200
kubectl port-forward service/vault 8200 &
vault kv put secret/accounts/aws AWS_SECRET_ACCESS_KEY=s3cr3t
Now we start a container in the demo-app
namespace and we us the AWS_SECRET_ACCESS_KEY
variable from a secret stored in Vault.
nano 05_demo.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: demo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-secrets
namespace: demo-app
spec:
replicas: 1
selector:
matchLabels:
app: hello-secrets
template:
metadata:
labels:
app: hello-secrets
annotations:
vault.security.banzaicloud.io/vault-addr: "https://vault.vault:8200"
vault.security.banzaicloud.io/vault-tls-secret: "vault-tls"
spec:
serviceAccountName: default
containers:
- name: nginx
image: nginxinc/nginx-unprivileged
command: ["sh", "-c", "echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000"]
env:
- name: AWS_SECRET_ACCESS_KEY
value: "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"
kubectl apply -f 05_demo.yaml
kubectl logs hello-secrets-676b67c659-fvk9d -n demo-app
time="2021-04-05T08:45:11Z" level=info msg="received new Vault token" app=vault-env
time="2021-04-05T08:45:11Z" level=info msg="initial Vault token arrived" app=vault-env
time="2021-04-05T08:45:11Z" level=info msg="spawning process: [sh -c echo $AWS_SECRET_ACCESS_KEY && echo going to sleep... && sleep 10000]" app=vault-env
time="2021-04-05T08:45:11Z" level=info msg="renewed Vault token" app=vault-env ttl=1h0m0s
s3cr3t
going to sleep...