Kubernetes integration with external Vault
In this post I will show you how you can integrate an external HashiCorp Vault to Kubernetes.
Parts of the K8S Security Lab series
Container Runetime Security
- Part1: How to deploy CRI-O with Firecracker?
- Part2: How to deploy CRI-O with gVisor?
- Part3: How to deploy containerd with Firecracker?
- Part4: How to deploy containerd with gVisor?
- Part5: How to deploy containerd with kata containers?
Advanced Kernel Security
- Part1: Hardening Kubernetes with seccomp
- Part2: Linux user namespace management wit CRI-O in Kubernetes
- Part3: Hardening Kubernetes with seccomp
Network Security
- Part1: RKE2 Install With Calico
- Part2: RKE2 Install With Cilium
- Part3: CNI-Genie: network separation with multiple CNI
- Part3: Configurre network wit nmstate operator
- Part3: Kubernetes Network Policy
- Part4: Kubernetes with external Ingress Controller with vxlan
- Part4: Kubernetes with external Ingress Controller with bgp
- Part4: Central authentication with oauth2-proxy
- Part5: Secure your applications with Pomerium Ingress Controller
- Part6: CrowdSec Intrusion Detection System (IDS) for Kubernetes
- Part7: Kubernetes audit logs and Falco
Secure Kubernetes Install
- Part1: Best Practices to keeping Kubernetes Clusters Secure
- Part2: Kubernetes Secure Install
- Part3: Kubernetes Hardening Guide with CIS 1.6 Benchmark
- Part4: Kubernetes Certificate Rotation
User Security
- Part1: How to create kubeconfig?
- Part2: How to create Users in Kubernetes the right way?
- Part3: Kubernetes Single Sign-on with Pinniped OpenID Connect
- Part4: Kubectl authentication with Kuberos Depricated !!
- Part5: Kubernetes authentication with Keycloak and gangway Depricated !!
- Part6: kube-openid-connect 1.0 Depricated !!
Image Security
Pod Security
- Part1: Using Admission Controllers
- Part2: RKE2 Pod Security Policy
- Part3: Kubernetes Pod Security Admission
- Part4: Kubernetes: How to migrate Pod Security Policy to Pod Security Admission?
- Part5: Pod Security Standards using Kyverno
- Part6: Kubernetes Cluster Policy with Kyverno
Secret Security
- Part1: Kubernetes and Vault integration
- Part2: Kubernetes External Vault integration
- Part3: ArgoCD and kubeseal to encript secrets
- Part4: Flux2 and kubeseal to encrypt secrets
- Part5: Flux2 and Mozilla SOPS to encrypt secrets
Monitoring and Observability
- Part6: K8S Logging And Monitoring
- Part7: Install Grafana Loki with Helm3
Backup
Vhat is Hashicorp Vault
HashiCorp Vault is a secrets management solution that brokers access for both humans and machines, through programmatic access, to systems. Secrets can be stored, dynamically generated, and in the case of encryption, keys can be consumed as a service without the need to expose the underlying key materials.
K3s install
ssh-copy-id vagrant@172.17.8.101
k3sup install \
--ip=172.17.8.101 \
--user=vagrant \
--sudo \
--tls-san=172.17.8.100 \
--cluster \
--k3s-channel=stable \
--k3s-extra-args "--no-deploy=traefik --flannel-iface=enp0s8 --node-ip=172.17.8.101" \
--merge \
--local-path $HOME/.kube/config \
--context=k3s-ha
Install vault
sudo dnf install -y dnf-plugins-core nano jq
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install -y vault
nano /etc/vault.d/vault.hcl
# HTTP listener
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = 1
}
# HTTPS listener
#listener "tcp" {
# address = "0.0.0.0:8200"
# tls_cert_file = "/opt/vault/tls/tls.crt"
# tls_key_file = "/opt/vault/tls/tls.key"
#}
api_addr = "http://0.0.0.0:8200"
systemctl start vault
systemctl enable vault
vault -autocomplete-install
complete -C /usr/bin/vault vault
export VAULT_ADDR=http://127.0.0.1:8200
echo "export VAULT_ADDR=http://127.0.0.1:8200" >> ~/.bashrc
vault status
vault operator init | tee /opt/vault/init.txt
Unseal Key 1: t4PsGsw8cj25l9tSpvh2Avr5647HhdaI27aAzSiYJz0=
Initial Root Token: s.sPKauYvv9iFKliclTIaMgbU1
export VAULT_TOKEN="s.sPKauYvv9iFKliclTIaMgbU1"
vault operator unseal t4PsGsw8cj25l9tSpvh2Avr5647HhdaI27aAzSiYJz0=
vault auth enable userpass
vault write auth/userpass/users/devopstales \
password=Password1 \
policies=admins
Integrate a Kubernetes Cluster with an External Vault
vault secrets enable kv
vault kv put kv/secret/devwebapp/config username='giraffe' password='salsa'
vault kv get -format=json kv/secret/devwebapp/config | jq ".data"
# create policy
vault policy write devwebapp-kv-ro - <<EOF
path "kv/secret/devwebapp/*" {
capabilities = ["read", "list"]
}
EOF
# create Kubernetes ServiceAccount
cat > internal-app.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: devwebapp
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: devwebapp
namespace: default
EOF
kubectl apply --filename internal-app.yaml
export EXTERNAL_VAULT_ADDR=172.17.8.101
export K8S_HOST=172.17.8.101
export VAULT_SA_NAME=$(kubectl get sa devwebapp \
-o jsonpath="{.secrets[*]['name']}")
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME \
-o jsonpath="{.data.token}" | base64 --decode; echo)
export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME \
-o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
vault auth enable kubernetes
vault write auth/kubernetes/config \
issuer="https://kubernetes.default.svc.cluster.local" \
token_reviewer_jwt="$SA_JWT_TOKEN" \
kubernetes_host="https://$K8S_HOST:6443" \
kubernetes_ca_cert="$SA_CA_CRT"
vault write auth/kubernetes/role/devwebapp \
bound_service_account_names=devwebapp \
bound_service_account_namespaces=default \
policies=devwebapp-kv-ro \
ttl=24h
INstall Vault Agent Injector
dnf copr enable cerenit/helm -y
dnf install helm -y
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
### változók???
helm install vault hashicorp/vault \
--set "injector.externalVaultAddr=http://$EXTERNAL_VAULT_ADDR:8200"
Demo
cat > devwebapp.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: orgchart
labels:
app: orgchart
spec:
selector:
matchLabels:
app: orgchart
replicas: 1
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "devwebapp"
vault.hashicorp.com/agent-inject-secret-config.txt: "kv/secret/devwebapp/config"
labels:
app: orgchart
spec:
serviceAccountName: devwebapp
containers:
- name: orgchart
image: jweissig/app:0.0.1
EOF
kubectl apply -f devwebapp.yaml
kubectl exec \
$(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") \
--container orgchart -- cat /vault/secrets/config.txt