trivy-operator 2.4: Patch release for Admisssion controller

Today I am happy to announce the release of trivy-operator 2.4. This blog post focuses on the functionality provided by the trivy-operator 2.4 release.

What is trivy-operator?

Trivy-operator is a Kubernetes Operator based on the open-source container vulnerability scanner Trivy. The goal of this project is to provide a vulnerability scanner that continuously scans containers deployed in a Kubernetes cluster. Built with Kubernetes Operator Pythonic Framework (Kopf) There are a few solution for checking the images when you deploy them to the Kubernetes cluster, but fighting against vulnerabilities is a day to day task. Check once is not enough when every day is a new das for frats. That is why I created trivy-operator so you can create scheduled image scans on your running pods.

What is new

With the release of trivy-operator 2.4 ther is the fallowin new features:

  • add policyreport creation and Policy Reporter UI integration
  • update vulnerabilityreports if exists
  • add ownerReferences for VulnerabilityReport
  • add redis cache
  • add documentation website

PolicyReport

The PolicyReport object is a prototype object probosed by the Kubernetes policy work group. The Policy Report Custom Resource Definition (CRD) can be used as a common way to provide policy results to Kubernetes cluster administrators and users, using native tools. See the proposal for background and details.

Add the PolicyReport CRDs to your cluster (v1alpha2):

kubectl create -f https://github.com/kubernetes-sigs/wg-policy-prototypes/raw/master/policy-report/crd/v1alpha2/wgpolicyk8s.io_policyreports.yaml

If you installed the trivy-operator by the helm chart the Policy Report Custom Resource Definition is installed automticle.

This objects can be visualized by the Policy Reporter UI.

Policy Reporter UI integration

The Policy Reporter UI is a monitoring and Observability Tool for the PolicyReport CRD with an optional UI. It is created by Kyverno. The main goal was a tool to visualize the resoluts of the Kyverno policies, but because it uses the PolicyReports CRD it can visualize the resoults of the trivy-operator scans.

Install the Policy Reporter UI:

helm repo add policy-reporter https://kyverno.github.io/policy-reporter
helm repo update

helm install policy-reporter policy-reporter/policy-reporter \
--set kyvernoPlugin.enabled=true --set ui.enabled=true --set ui.plugins.kyverno=true \
-n policy-reporter --create-namespace

kubectl port-forward service/policy-reporter-ui 8082:8080 -n policy-reporter

Open http://localhost:8082/ in your browser.

Policy Reporter UI

Trivy Image Validator

The admission controller function can be configured as a ValidatingWebhook in a k8s cluster. Kubernetes will send requests to the admission server when a Pod creation is initiated. The admission controller checks the image using trivy if it is in a namespace with the label trivy-operator-validation=true.

You can define policy to the Admission Controller, by adding annotation to the pod trough the deployment:

spec:
  ...
  template:
    metadata:
      annotations:
        trivy.security.devopstales.io/medium: "5"
        trivy.security.devopstales.io/low: "10"
        trivy.security.devopstales.io/critical: "2"
...

Where you can find:

With the release of trivy-operator 2.4 I published trivy-operator with OperatorFramework to OperatorHub:

OperatorHub

OperatorHub

Usage

To ease deployment I created a helm chart for trivy-operator.

helm repo add devopstales https://devopstales.github.io/helm-charts
helm repo update

Create a value file for deploy:

cat <<'EOF'> values.yaml
image:
  repository: devopstales/trivy-operator
  pullPolicy: Always
  tag: "2.3"

imagePullSecrets: []
podSecurityContext:
  fsGroup: 10001
  fsGroupChangePolicy: "OnRootMismatch"

serviceAccount:
  create: true
  annotations: {}
  name: "trivy-operator"

monitoring:
  port: "9115"

serviceMonitor:
  enabled: false
  namespace: "kube-system"

storage:
  enabled: true
  size: 1Gi

NamespaceScanner:
  crontab: "*/5 * * * *"
  namespaceSelector: "trivy-scan"

registryAuth:
  enabled: false
  registry:
  - name: docker.io
    user: "user"
    password: "password"

githubToken:
  enabled: false
  token: ""
EOF

When the trivy in the container want to scan an image first download the vulnerability database from github. If you test many images you need a githubToken overcome the github rate limit and dockerhub username and password for overcome the dockerhub rate limit. If your store you images in a private repository you need to add an username and password for authentication.

The following tables lists configurable parameters of the trivy-operator chart and their default values.

Parameter Description Default
image.repository image devopstales/trivy-operator
image.pullPolicy pullPolicy Always
image.tag image tag 2.1
imagePullSecrets imagePullSecrets list []
podSecurityContext.fsGroup mount id 10001
serviceAccount.create create serviceAccount true
serviceAccount.annotations add annotation to serviceAccount {}
serviceAccount.name name of the serviceAccount trivy-operator
monitoring.port prometheus endpoint port 9115
serviceMonitor.enabled enable serviceMonitor object creation false
serviceMonitor.namespace where to create serviceMonitor object kube-system
storage.enabled enable pv to store trivy database true
storage.size pv size 1Gi
NamespaceScanner.crontab cronjob scheduler “*/5 * * * *”
NamespaceScanner.namespaceSelector Namespace Selector “trivy-scan”
registryAuth.enabled enable registry authentication in operator false
registryAuth.registry registry name for authentication
registryAuth.user username for authentication
registryAuth.password password for authentication
githubToken.enabled Enable githubToken usage for trivy database update false
githubToken.token githubToken value ""
kubectl create ns trivy-operator
kubens trivy-operator
helm upgrade --install trivy devopstales/trivy-operator -f values.yaml

Monitoring

Trivy-operatos has a prometheus endpoint op port 9115 and can be deployed wit ServiceMonitor for automated scrapping.

curl -s http://10.43.179.39:9115/metrics | grep trivy_vulnerabilities
# HELP trivy_vulnerabilities_sum Container vulnerabilities
# TYPE trivy_vulnerabilities_sum gauge
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/openshift/mysql-56-centos7:latest",severity="scanning_error"} 1.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",severity="UNKNOWN"} 0.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",severity="LOW"} 83.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",severity="MEDIUM"} 5.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",severity="HIGH"} 7.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",severity="CRITICAL"} 4.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/library/nginx:1.18",severity="UNKNOWN"} 0.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/library/nginx:1.18",severity="LOW"} 126.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/library/nginx:1.18",severity="MEDIUM"} 25.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/library/nginx:1.18",severity="HIGH"} 43.0
trivy_vulnerabilities_sum{exported_namespace="trivytest",image="docker.io/library/nginx:1.18",severity="CRITICAL"} 21.0
# HELP trivy_vulnerabilities Container vulnerabilities
# TYPE trivy_vulnerabilities gauge
trivy_vulnerabilities{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",installedVersion="2.3.4",pkgName="pkgName",severity="LOW",vulnerabilityId="CVE-2011-3374"} 1.0
trivy_vulnerabilities{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",installedVersion="8.32-4",pkgName="pkgName",severity="LOW",vulnerabilityId="CVE-2016-2781"} 1.0
trivy_vulnerabilities{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",installedVersion="8.32-4",pkgName="pkgName",severity="LOW",vulnerabilityId="CVE-2017-18018"} 1.0
trivy_vulnerabilities{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",installedVersion="7.74.0-1.3",pkgName="pkgName",severity="CRITICAL",vulnerabilityId="CVE-2021-22945"} 1.0
trivy_vulnerabilities{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",installedVersion="7.74.0-1.3",pkgName="pkgName",severity="HIGH",vulnerabilityId="CVE-2021-22946"} 1.0
trivy_vulnerabilities{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",installedVersion="7.74.0-1.3",pkgName="pkgName",severity="MEDIUM",vulnerabilityId="CVE-2021-22947"} 1.0
trivy_vulnerabilities{exported_namespace="trivytest",image="docker.io/nginxinc/nginx-unprivileged:latest",installedVersion="7.74.0-1.3",pkgName="pkgName",severity="LOW",vulnerabilityId="CVE-2021-22898"} 1.0

Grafana Dashboard