ArgoCD and kubeseal to encrypt secrets

Kubernetes
Page content

In this post I will show you how you can use kubeseal with ArgoCD to protect secrets.

Parts of the K8S Gitops series

Install Argocd

kubectl create namespace argocd
kubectl apply -n argocd -f \
https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# or in ha
kubectl apply -n argocd -f \
https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml

Install Argocd cli

VERSION=$(curl --silent "https://api.github.com/repos/argoproj/argo-cd/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')

curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64

chmod +x /usr/local/bin/argocd

argocd version

Connect without ingress

kubectl port-forward svc/argocd-server -n argocd 8080:443

Create ingress for server

---
piVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: argocd-server-http-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    cert-manager.io/cluster-issuer: ca-issuer
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: argocd-server
          servicePort: http
    host: argocd.k8s.intra
  tls:
  - hosts:
    - argocd.k8s.intra
    secretName: https-argocd-secret # do not change, this is provided by Argo CD
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: argocd-server-grpc-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: argocd-server
          servicePort: https
    host: grpc-argocd.k8s.intra
  tls:
  - hosts:
    - grpc-argocd.k8s.intra
    secretName: argocd-secret # do not change, this is provided by Argo CD

Get init password

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d && echo
jMnyrjcdocMoqPfC

argocd login argocd.k8s.intra
WARNING: server certificate had error: x509: certificate signed by unknown authority. Proceed insecurely (y/n)? y
WARN[0002] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
Username: admin
Password: 
'admin:login' logged in successfully
Context 'argocd.k8s.intra' updated

argocd account update-password

Register new cluster

By default Argocd register the cluster where running with a cluster admin service account. You can register different clusters with its kubectl configs. So you can create multiple service accounts with multiple privileges and register them with its kubectl configs.

$ kubectx
default

$ argocd cluster add default
INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system" 
INFO[0000] ClusterRole "argocd-manager-role" updated    
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated 
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web. 
Cluster 'https://172.17.9.10:6443' added

Deploy app with Argocd

$ kubectl create ns guestbook-demo

$ argocd app create 00-tools --repo https://github.com/devopstales/gitops-repo.git \
--path 00_argocd/00_tools --dest-server https://kubernetes.default.svc --dest-namespace default

$ argocd app create 01-guestbook --repo https://github.com/devopstales/gitops-repo.git \
--path 00_argocd/01_guestbook --dest-server https://kubernetes.default.svc --dest-namespace guestbook-demo

$ argocd app get 01-guestbook
Name:               01-guestbook
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          guestbook-demo
URL:                https://argocd.k8s.intra/applications/01-guestbook
Repo:               https://github.com/devopstales/gitops-repo.git
Target:             
Path:               00_argocd/01_guestbook
SyncWindow:         Sync Allowed
Sync Policy:        <none>
Sync Status:        OutOfSync from  (e8df0a5)
Health Status:      Missing

GROUP                      KIND         NAMESPACE       NAME                            STATUS     HEALTH   HOOK  MESSAGE
                           Service      guestbook-demo  guestbook-ui                    OutOfSync  Missing        
apps                       Deployment   guestbook-demo  guestbook-ui                    OutOfSync  Missing        
rbac.authorization.k8s.io  RoleBinding  guestbook-demo  psp-rolebinding-guestbook-demo  OutOfSync  Missing   

$ argocd app sync 01-guestbook

Install kubeseal

VERSION=$(curl --silent "https://api.github.com/repos/bitnami-labs/sealed-secrets/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/$VERSION/kubeseal-linux-amd64 -O /usr/local/bin/kubeseal
chmod 755 /usr/local/bin/kubeseal
kubeseal --version

Now install the cluster-side controller

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/$VERSION/controller.yaml

Create a sealed secret

Create a secret you want to encrypt:

apiVersion: v1
data:
  secret: UzNDUjNUCg==
kind: Secret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: demo-app

A secret in Kubernetes cluster is encoded in base64 but not encrypted! Theses data are “only” encoded so if a user have access to your secrets, he can simply base64 decode to see your sensitive data:

echo "UzNDUjNUCg==" | base64 -d
S3CR3T

Since the secrets aren’t encrypted, it is unsecure to commit them to your Git repository.

Use kubeseal to Encrypt the secret

kubeseal --format yaml <secret.yaml >sealedsecret.yaml

cat sealedsecret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: demo-app
spec:
  encryptedData:
    secret: AgCZl5b75Lmr8z7Ppa5tNBPX4zWH2vset0GVKhNfTBRnAANDs9Gycrq5EfueE00PGX++v4VFKEwi9rNeAAvFkETges31Uhi4+Oym9CkV9rU2pHAvD4iZapt+fHSndMUY8vWT8GCYzrzSOFSRPKB4cdAy3JJ4f48SwxCFYXdJgl/6KiHkrk2AzxHKip3ryVjKY01E8cSpxw1Exv8RnEDD8D9hfb57fEIRRwMrIRUkg/jPOvf4YCHcjHiVLLP+MwutT1Jd65hjAx1WZFSjDRUj3rFfzsO6zAVxgx20WXtc3qMK9jMeeQaNbbAvdv3YuNsuxJIE8SFQFPfGop+QFefiyDGWTjzwHkeU65Ci1Nuj8pSS600ITyGdyNY4F3qjen1eBnMOaub5ZJqEmXyTQwSL/9R7UfoFqJCo4b36g2axacegqHtLL+U4wrHsDB9iQ/JrEAWj4l7s5bhOJbq0N8zLwZvEGXSoPs/4eBUxCuHayOCz6o8BY8Zsv1tDgQ+AXpvudXfzw02zH/DCr7Jg2CVXB8Qk2SUnC5rMzsvqcsYnHP25pxGh9qd3p8QXIjb+AttJUFkPGHlc/rY6sY4QJ6Qjlfv8VXArwrmnfkcZSfLDwyUOGcqZiho3+vGC4mjDcFgbEDbD3Emv/2jHimFBOv2eq9dMqvmZuzk4M4KCLYHqFuX+L/XM+mAnAxlCRrv6q6Hup26HuI84Hn2N
  template:
    metadata:
      creationTimestamp: null
      name: mysecret
      namespace: demo-app

sealedsecret.yaml is the file you need to store in git.

argocd app create 02-secret --repo https://github.com/devopstales/gitops-repo.git \
--path 00_argocd/02_secret --dest-server https://kubernetes.default.svc --dest-namespace demo-app

kubectl logs -n demo-app demo-app
S3CR3T