LabHub

Blog

Kyverno Image Verification: Sigstore and Supply Chain Security

2026-03-20 · 15 min read · #kyverno#policy#security#open-source-internals

한국어English日本語

1. verifyImages Rule Overview

Kyverno's verifyImages rule verifies container image signatures and attestations to strengthen supply chain security.

1.1 Basic Structure

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-images
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: verify-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - 'ghcr.io/myorg/*'
            - 'myregistry.io/apps/*'
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                      -----END PUBLIC KEY-----

2. Cosign Signature Verification

2.1 Static Key

verifyImages:
  - imageReferences:
      - 'ghcr.io/myorg/*'
    attestors:
      - entries:
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                -----END PUBLIC KEY-----

2.2 Keyless Signing (Fulcio)

OIDC-based keyless signature verification:

verifyImages:
  - imageReferences:
      - 'ghcr.io/myorg/*'
    attestors:
      - entries:
          - keyless:
              url: https://fulcio.sigstore.dev
              rekor:
                url: https://rekor.sigstore.dev
              subject: 'https://github.com/myorg/*'
              issuer: 'https://token.actions.githubusercontent.com'

This policy:

2.3 KMS Keys

verifyImages:
  - imageReferences:
      - 'myregistry.io/apps/*'
    attestors:
      - entries:
          - keys:
              kms: awskms:///arn:aws:kms:us-east-1:123456789:key/abc-123
          # or
          - keys:
              kms: gcpkms://projects/my-project/locations/global/keyRings/my-ring/cryptoKeys/my-key
          # or
          - keys:
              kms: azurekms://my-vault.vault.azure.net/keys/my-key

3. Attestation Verification

3.1 in-toto Attestation

verifyImages:
  - imageReferences:
      - 'ghcr.io/myorg/*'
    attestations:
      - type: https://slsa.dev/provenance/v1
        attestors:
          - entries:
              - keyless:
                  url: https://fulcio.sigstore.dev
                  rekor:
                    url: https://rekor.sigstore.dev
        conditions:
          - all:
              - key: '{{ builder.id }}'
                operator: Equals
                value: 'https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.9.0'

3.2 SLSA Provenance

verifyImages:
  - imageReferences:
      - 'ghcr.io/myorg/*'
    attestations:
      - type: https://slsa.dev/provenance/v1
        attestors:
          - entries:
              - keyless:
                  url: https://fulcio.sigstore.dev
                  subject: 'https://github.com/myorg/*'
                  issuer: 'https://token.actions.githubusercontent.com'
        conditions:
          - all:
              - key: '{{ buildDefinition.buildType }}'
                operator: Equals
                value: 'https://slsa-framework.github.io/github-actions-buildtypes/workflow/v1'
              - key: '{{ runDetails.builder.id }}'
                operator: Equals
                value: 'https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.9.0'

4. Image Registry Authentication

4.1 Private Registry Access

# For Kyverno to look up signatures in a private registry, the registry
# credentials must reach Kyverno itself — via imagePullSecrets or the
# ServiceAccount.

# values.yaml (Helm install)
# admissionController:
#   container:
#     image:
#       pullSecrets:
#         - name: my-registry-secret

5. SBOM Verification

5.1 CycloneDX SBOM Attestation

verifyImages:
  - imageReferences:
      - 'ghcr.io/myorg/*'
    attestations:
      - type: https://cyclonedx.org/bom/v1.4
        attestors:
          - entries:
              - keyless:
                  url: https://fulcio.sigstore.dev
        conditions:
          - all:
              - key: "{{ components[?name=='log4j-core'].version | [0] }}"
                operator: NotEquals
                value: '2.14.1'

6. Image Mutation

6.1 Converting Tags to Digests

verifyImages:
  - imageReferences:
      - 'ghcr.io/myorg/*'
    mutateDigest: true # Auto-convert tags to digests
    required: true # Signature must exist
    verifyDigest: true # Verify digest
    attestors:
      - entries:
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                ...
                -----END PUBLIC KEY-----

mutateDigest: true rewrites the image tag into a SHA256 digest automatically, guaranteeing image immutability.

7. Worked Policy: Comprehensive Image Security

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: comprehensive-image-security
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    # 1. Only approved registries
    - name: allowed-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: 'Images must be from approved registries'
        foreach:
          - list: 'request.object.spec.[initContainers, containers][]'
            deny:
              conditions:
                all:
                  - key: '{{ element.image }}'
                    operator: AnyNotIn
                    value:
                      - 'ghcr.io/myorg/*'
                      - 'myregistry.io/*'

    # 2. No latest tag
    - name: deny-latest
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Using 'latest' tag is not allowed"
        foreach:
          - list: 'request.object.spec.[initContainers, containers][]'
            deny:
              conditions:
                any:
                  - key: '{{ element.image }}'
                    operator: Equals
                    value: '*:latest'

    # 3. Signature verification
    - name: verify-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - 'ghcr.io/myorg/*'
          mutateDigest: true
          required: true
          attestors:
            - entries:
                - keyless:
                    url: https://fulcio.sigstore.dev
                    subject: 'https://github.com/myorg/*'
                    issuer: 'https://token.actions.githubusercontent.com'

8. How Verification Actually Runs at Admission

The YAML above only declares what to check. The work happens the moment an admission request arrives. When a Pod creation request reaches the API server, the Kyverno admission controller pulls every container image reference out of the Pod spec, keeps the ones matching an imageReferences pattern, and asks the registry for their signatures and attestations. The fact that is easy to miss is that this lookup is a network call leaving the cluster. From the moment the policy is enabled, the registry is not only the path images are pulled from — it is also the path that decides whether a Pod can exist at all.

Signatures do not always live in the same repository as the image. If mirroring or an internal rule keeps signatures somewhere else, repository changes where they are fetched from. When there is more than one attestor, attestors.count decides how many of them have to pass. The docs state that when the value is not specified, all attestors are verified. That difference shows up during key rotation. Put the old and the new key in entries and set the count to 1, and an image signed with either key passes, so the rotation window costs no downtime. Leave the count out and only images signed with both keys pass — the exact opposite result.

verifyImages:
  - imageReferences:
      - 'ghcr.io/myorg/*'
    skipImageReferences:
      - 'ghcr.io/myorg/legacy-*'
    repository: 'ghcr.io/myorg/signatures' # fetch signatures from another repo
    required: true
    verifyDigest: true
    mutateDigest: true
    attestors:
      - count: 1 # only 1 of the entries must verify (key rotation window)
        entries:
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                (current key)
                -----END PUBLIC KEY-----
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                (incoming key)
                -----END PUBLIC KEY-----

If every repeated pull of the same image paid another registry round trip, admission latency would become deployment latency. Kyverno keeps verification results in a TTL cache to avoid that. These are install-level settings, not policy fields, and the defaults are cache enabled true, a maximum of 1000 keys, and a TTL of 60m. Passing 0 for the size or the TTL resets it to the default. Thanks to that cache, a twenty-replica rollout really pays the registry round trip only once. It is also why a second performance measurement comes out far faster than the first, and why the first deployment after the TTL expires is the slow one. Turn the cache off and measure admission latency, and you get something close to the worst case you will see when the registry is down.

# Kyverno install-level settings — not policy fields
imageVerifyCacheEnabled: true # default true
imageVerifyCacheMaxSize: 1000 # default 1000, 0 resets to default
imageVerifyCacheTTLDuration: 60m # default 60m, 0 resets to default

The webhookTimeoutSeconds: 30 in the first example of this post is not an accident. The field is the maximum time allowed to apply that policy; the documented default is 10s and the value must be between 1 and 30 seconds. A policy that makes a registry round trip will eventually outgrow the default 10s. What happens on timeout is decided by failurePolicy: the default Fail rejects the request, Ignore lets it through unverified. Whether a slow registry stops every deployment or silently removes verification comes down to that one field. Both fields are marked deprecated as of 1.13 and move to webhookConfiguration.timeoutSeconds and webhookConfiguration.failurePolicy, so check the docs for the version you run to see which one is actually read.

The effect of mutateDigest: true on a rollout is visible the day you enable it. The policy rewrites the tag into a digest, so once the Deployment is applied what stays in the Pod spec is not a tag but a pinned reference starting with @sha256:. Overwriting the same tag in the registry no longer changes anything — not for running Pods, and not for Pods created later by a scale-out, which still use the originally pinned image. That is the immutability you asked for, but a pipeline that used to push a tag and delete Pods to pick up the new image now quietly does nothing at all. That is usually the first question that arrives the day after mutateDigest goes on.

The remaining three fields have clear roles. required enforces that all matching images were verified, verifyDigest enforces that digests are used at all, and skipImageReferences is the list of patterns excluded from matching. Rather than cloning a whole policy to carve out an exception, write it here. And imageReferences does not take variable interpolation — the docs are explicit about it. A policy that varies the allowed registry by namespace label cannot be built from this field alone; split the policy or handle it with a separate validate rule.

9. End to End: Sign It, Enforce It, Watch It Get Blocked

Start locally, not in the cluster. Generate a key pair with cosign, sign the image, and verify it by hand with the public half of the same key. If it fails here, no amount of policy editing will make it pass. If attestations are part of the plan, attach the predicate file with attest and confirm it with verify-attestation before going anywhere near the cluster.

cosign generate-key-pair
cosign sign --key cosign.key ${IMAGE}
cosign verify --key cosign.pub ${IMAGE}

# if attestations are also in scope
cosign attest --key cosign.key --predicate <file> --type <predicate type> ${IMAGE}
cosign verify-attestation --key cosign.pub --type <type> ${IMAGE}

The next step is checking the policy before it reaches the cluster. kyverno apply runs policies and resources against each other locally. An image verification policy is only meaningful if it can reach the registry, so add --registry, which the docs describe as using local docker credentials to reach the image registry. Use -t for a tabular result, --detailed-results to expand down to individual rules, and -p for report-shaped output. If failures or errors are found the command exits 1, so wiring it into a CI job stops the pipeline the moment an unsigned image lands in a manifest. Catching it before the policy is deployed is worth a great deal more than catching it after.

kyverno apply policy.yaml --resource pod.yaml --registry
kyverno apply policy.yaml --resource pod.yaml --registry -t
kyverno apply policy.yaml --resource pod.yaml --registry --detailed-results
kyverno apply policy.yaml --resource pod.yaml --registry --policy-report

# 1 if failures or errors are found
echo $?

Once local results match expectations, apply the policy and deliberately push an unsigned image to see the block happen. The rejection comes back as the API server relaying the webhook response, and the message names the policy and rule that blocked it along with the reason. A missing signature produces a signature not found style message; a signature that exists but does not match the key produces an invalid signature style message. Telling those two apart is where the next section starts.

kubectl apply -f policy.yaml
kubectl get cpol,pol -A
kubectl apply -f unsigned-pod.yaml

10. Failure Cases and the Order to Diagnose Them

The symptom almost always collapses to one sentence: the Pod will not start. The causes fan out into five branches, and skipping the order means digging in the wrong place for a long time. Diagnose from the outside in — first confirm the policy is running at all, then work down to the signature itself.

The first check is whether the policy is being evaluated. A Pod that passed does not prove the policy works; it may prove the policy is not running. Look at the Pod status, confirm the ready column of the policy list is true everywhere, and confirm the webhooks are registered. Kyverno registers as two types of webhooks, and an image verification policy using mutateDigest also hangs off the mutating side.

kubectl -n kyverno get po
kubectl get cpol,pol -A
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

The second is separating a missing signature from a wrong one. Run the cosign verify from the previous section by hand. If it fails by hand the problem is the build pipeline, not the cluster; if it passes by hand and only fails in-cluster, that is when you start looking at Kyverno. This single check filters out half the cases.

The third is registry authentication. If Kyverno itself cannot read a private registry, a perfectly valid signature looks identical to a missing one. The trap is that a Pod pulling images successfully and Kyverno fetching signatures are completely separate credential paths. The docs walk through creating a docker-registry secret in the kyverno namespace and passing it to the Kyverno deployment with --imagePullSecrets. To use different credentials per policy there is imageRegistryCredentials. If the registry is signed by an internal CA there is a separate trust problem, and the docs offer two answers: replace the certificate store with global.caCertificates.data, or mount host certificates with global.caCertificates.volume.

The fourth is an attestor mismatch. The signature exists and can be fetched, but verification still fails — most often with keyless. The subject and issuer have to match the workflow path and tag exactly, so renaming a release workflow file or changing a tag convention blocks everything from that moment on. And check again whether attestors.count was specified: without it, every entry must verify. If adding one key suddenly made everything fail, that is almost certainly why.

The fifth is Rekor being unreachable. In an air-gapped environment, or behind an outbound proxy, a blocked transparency-log lookup stops verification even with a flawless signature and key. Setting ignoreTlog to true under rekor skips transparency log verification, and ignoreSCT to true under ctlog skips SCT verification. These are not workarounds but a deliberate reduction in verification scope, so leave a note in the policy about what was given up.

If none of that catches it, raise the log level. -v=4 shows variable substitution, -v=6 is the highest verbosity, and dumpPayload=true prints the full AdmissionReview contents. The last one is loud enough that it should be turned on right before reproducing and off right after. And if a policy ends up blocking the API server itself so that nothing can be deployed, scale the controller to zero or delete the webhook configurations to bring the cluster back first, then look for the cause. Both commands switch off policy enforcement across the cluster, so they must be reverted once you are back on your feet.

kubectl -n kyverno edit deploy kyverno-admission-controller
kubectl -n kyverno logs <pod_name> -f

# last resort — policy enforcement is off cluster-wide while this stands
kubectl scale deploy kyverno-admission-controller -n kyverno --replicas 0
kubectl delete validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg
kubectl delete mutatingwebhookconfiguration kyverno-resource-mutating-webhook-cfg

11. When Not to Use This

If a mirror registry received the images but not the signature artifacts, this policy fails on everything, without exception. It is the most common thing that goes wrong when images are imported into an air-gapped environment. The image is in the internal registry while the signature stayed in the original one, so until repository points at where the signatures actually live, or the mirroring pipeline is fixed to carry them across, Enforce must stay off. Add a keyless setup that also needs a Rekor lookup and the whole arrangement simply does not hold in an environment with no outbound access.

It is also hard to apply to images you did not build. Third-party images are frequently unsigned, and when they are signed it is with a key you have no trust relationship with. The answer becomes skipImageReferences or splitting the policy, but once the exception list covers more than half of the images actually running, the policy no longer provides security — it provides the impression of security. At that point it is more honest to keep the allowed-registry check and apply signature verification only to internal build output.

Starting a development cluster in Enforce mode is another thing to avoid. Turning on enforcement before the signing pipeline has reached every team blocks deployments outright, and the bottleneck ends up being the person who wrote the policy rather than the policy itself. Run it in Audit for a few days, read the reports, and promote to Enforce when the list of blocked images matches what you expected.

Finally, making the registry a dependency of the admission path is a cost in itself. With the default failurePolicy of Fail, Pod creation stalls while the registry is slow; with Ignore, there is effectively no verification during that window. The docs note that Ignore keeps registry failures from blocking operations and is useful when images already exist on nodes — but that also means unsigned images can get in during the outage. Neither option is free, and this choice belongs with whoever owns service availability, not only with the security team.

12. References

13. Summary

  1. cosign verification: Static key, Keyless (Fulcio), KMS support
  2. Attestation verification: in-toto, SLSA provenance condition-based verification
  3. SBOM verification: Check vulnerable components in CycloneDX/SPDX attestations
  4. Digest mutation: Ensure image immutability with mutateDigest
  5. Comprehensive policies: Combine registry restrictions + tag policies + signature verification
  6. Operational parameters: the verification cache, webhookTimeoutSeconds and failurePolicy decide admission latency and outage behaviour
  7. Diagnosis order: policy is running, signature exists, registry auth, attestor match, Rekor reachability

The next post covers a comparison between Kyverno and OPA/Gatekeeper.

Comments

No comments yet.

Sign in to leave a comment