Image Signature Verification with Kyverno
In this post I will show you how you can use Kyverno and Cosign for Image Signature Verification in a Kubernetes cluster.
Parst of the K8S Security series
- Part1: Best Practices to keeping Kubernetes Clusters Secure
- Part2: Kubernetes Hardening Guide with CIS 1.6 Benchmark
- Part3: RKE2 The Secure Kubernetes Engine
- Part4: RKE2 Install With cilium
- Part5: Kubernetes Certificate Rotation
- Part6: Hardening Kubernetes with seccomp
- Part7a: RKE2 Pod Security Policy
- Part7b: Kubernetes Pod Security Admission
- Part7c: Pod Security Standards using Kyverno
- Part8: Kubernetes Network Policy
- Part9: Kubernetes Cluster Policy with Kyverno
- Part10: Using Admission Controllers
- Part11a: Image security Admission Controller
- Part11b: Image security Admission Controller V2
- Part11c: Image security Admission Controller V3
- Part12: Continuous Image security
- Part13: K8S Logging And Monitoring
- Part14: Kubernetes audit logs and Falco
- Part15a Image Signature Verification with Connaisseur
- Part15b Image Signature Verification with Connaisseur 2.0
- Part15c Image Signature Verification with Kyverno
- Part16a Backup your Kubernetes Cluster
- Part16b How to Backup Kubernetes to git?
- Part17a Kubernetes and Vault integration
- Part17b Kubernetes External Vault integration
- Part18a: ArgoCD and kubeseal to encript secrets
- Part18b: Flux2 and kubeseal to encrypt secrets
- Part18c: Flux2 and Mozilla SOPS to encrypt secrets
- Part19: ArgoCD auto image updater
- Part20: Secure k3s with gVisor
- Part21: How to use imagePullSecrets cluster-wide??
- Part22: Automatically change registry in pod definition
Wat is Cosign?
Cosign is a new open-source tool to manage the process of signing and verifying container images. Developed by Googele in collaboration with Linux Foundation’s sigstore project. The motivation for cosign is “to make signatures invisible infrastructure.” With Images signed by Cosign you didn’t neet to change your infrastructure to store the public signing key, like Notary. (With Notary you need a Notary server connected to your registry to store the keys) With Cosign, the signatures directly appear as tags of the image linked to the associated image via the digest:
Key Management options:
- fixed, text-based keys generated using
cosign generate-key-pair
- cloud KMS-based keys generated using
cosign generate-key-pair -kms
- keys generated on hardware tokens using the PIV interface using
cosign piv-tool
- Kubernetes-secret based keys generated using
cosign generate-key-pair -k8s
Installing Cosign
It’s a golang project, so it’s fairly easy to get started, there’s a single binary available from their release pag and it has been signed by them.
Generate key pair with:
$ cosign generate-key-pair
Enter password for private key:
Enter again:
Private key written to cosign.key
Public key written to cosign.pub
Sign an image with cosign:
docker pull alpine:edge
docker tag alpine:edge devopstales/testimage:unsigned
docker push devopstales/testimage:unsigned
docker pull alpine:latest
docker tag alpine:latest devopstales/testimage:cosign
docker push devopstales/testimage:cosign
cosign sign -key ~/data/cosign.key devopstales/testimage:cosign
Enter password for private key:
Pushing signature to: index.docker.io/devopstales/testimage:sha256-4661fb57f7890b9145907a1fe2555091d333ff3d28db86c3bb906f6a2be93c87.sig
Verify a container against a public key:
$ cosign verify -key ~/data/cosign.pub devopstales/testimage:cosign
Verification for devopstales/testimage:cosign --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
- Any certificates were verified against the Fulcio roots.
{"critical":{"identity":{"docker-reference":"index.docker.io/devopstales/testimage"},"image":{"docker-manifest-digest":"sha256:4661fb57f7890b9145907a1fe2555091d333ff3d28db86c3bb906f6a2be93c87"},"type":"cosign container image signature"},"optional":null}
Image Signature Verification tools
In a previous post I used Connaisseur to Image Signature Verification. I could youse Connaisseur with Cosign too but with the new release of Kyverno we didn’t need to deploy a separate tool for Image Signature Verification. We can use Kyverno’s verifyImages
rule.
It validate signatures for matching images using Cosign and mutates image references with the digest returned by Cosign. Using an image digest guarantees immutability of images and hence improves security.
Install the latest version of Kyverno:
kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/main/definitions/release/install.yaml
Patch the Kyverno webhook, to allow time for calling the OCI registry:
kubectl patch mutatingwebhookconfigurations kyverno-resource-mutating-webhook-cfg \
--type json \
-p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"},{"op": "replace", "path": "/webhooks/0/timeoutSeconds", "value": 15}]'
Here is a policy that verifies all images from a specific repository:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: check-image
spec:
validationFailureAction: enforce
background: false
rules:
- name: check-image
match:
resources:
kinds:
- Pod
verifyImages:
- image: "docker.io/devopstales/testimage:*"
key: |-
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEL53O1V5FP2Vaa60BTwRjrOxhuu5C
iB/mODf/V2eiGw+WbA689ZZRjWwXCf+4jwzfRSrik0YvTCMqvl3BDaPG2A==
-----END PUBLIC KEY-----
kubectl create deployment signed-my \
--image=devopstales/testimage:cosign
Try running an unsigned image that matches the configured rule:
kubectl create deployment unsigned-my \
--image=docker.io/devopstales/testimage:unsigned
This will be blocked:
error: failed to create deployment: admission webhook "mutate.kyverno.svc" denied the request:
resource Deployment/kyverno-system/unsigned-my was blocked due to the following policies
verify-image:
autogen-verify-image: 'image verification failed for docker.io/devopstales/testimage:unsigned:
failed to verify image: fetching signatures: getting signature manifest: GET https://index.docker.io/v2/devopstales/testimage/manifests/sha256-0119f88f395766eb52f9b817c3d23576bf31935dc8e94abe14bae9a083ce4639.sig:
MANIFEST_UNKNOWN: manifest unknown; map[Tag:sha256-0119f88f395766eb52f9b817c3d23576bf31935dc8e94abe14bae9a083ce4639.sig]'