Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-67364

SignatureValidationFailed when creating a pod after applying an image policy with PKI.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.21
    • Node / CRI-O
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • Rejected
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          SignatureValidationFailed when creating a pod after applying an image policy with PKI. 
      The image used is signed with cosign PKI. (refer to https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-86101)

      Version-Release number of selected component (if applicable):

          4.21.0-0.nightly-2025-12-10-193741

      How reproducible:

          Always 

      Steps to Reproduce:

          1. Set crio log level as "debug"
      #oc create -f - << EOF
       apiVersion: machineconfiguration.openshift.io/v1
       kind: ContainerRuntimeConfig
       metadata:
        name: set-loglevel
       spec:
        machineConfigPoolSelector:
          matchLabels:
           pools.operator.machineconfiguration.openshift.io/worker: ""
        containerRuntimeConfig:
          logLevel: debug
       EOF
      
          2. create a user namespace
       % oc new-project testnamespace 
      
          3. create an imagepolicy crd
      apiVersion: config.openshift.io/v1
       kind: ImagePolicy
       metadata:
         name: byopkipolicy   # BYOPKI with Root CA and Intermediate CA
         namespace: testnamespace
       spec:
         scopes:
         - quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
         policy:
           rootOfTrust:
             policyType: PKI
             pki:
               caRootsData: LS0tLS1CRUdJT....Q0FURS0tLS0t
               caIntermediatesData: LS0tLS1CRUd....JQ0FURS0tLS0t
               pkiCertificateSubject:
                  email: minmli@redhat.com
                  hostname: myhost.example.com
      
            4. wait a few minutes, check the policy take effect on worker
      sh-5.1# cat /etc/crio/policies/testnamespace.json
      
      sh-5.1# cat /etc/containers/registries.d/sigstore-registries.yaml  
      docker:
         quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3:
           use-sigstore-attachments: true
      
            5. create a pod with the signed image
      #oc apply -f - << EOF
       apiVersion: v1
       kind: Pod
       metadata:
         name: testpod-1
         labels:
           app: testpod
       spec:
         securityContext:
           runAsNonRoot: true
           seccompProfile:
             type: RuntimeDefault
         containers:
         - name: hello-pod
           image: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
           securityContext:
             allowPrivilegeEscalation: false
             capabilities:
               drop: ["ALL"]
           ports:
           - containerPort: 80
       EOF
      
               

      Actual results:

          5 failed to pull image due to error : SignatureValidationFailed
      % oc get pod 
      NAME        READY   STATUS                      RESTARTS   AGE
      testpod-1   0/1     SignatureValidationFailed   0          5s
      
      % oc describe pod testpod-1
      ...
      Events:
        Type     Reason          Age               From     Message
        ----     ------          ----              ----     -------
        Normal   AddedInterface  24s               multus   Add eth0 [10.128.2.22/23] from ovn-kubernetes
        Normal   BackOff         23s               kubelet  Back-off pulling image "quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3"
        Warning  Failed          23s               kubelet  Error: ImagePullBackOff
        Normal   Pulling         9s (x2 over 24s)  kubelet  Pulling image "quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3"
        Warning  Failed          8s (x2 over 24s)  kubelet  Failed to pull image "quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3": SignatureValidationFailed: unable to pull image or OCI artifact: pull image err: Source image rejected: Signature for identity "quay.io/minmli/quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" is not accepted; artifact err: provided artifact is a container image
        Warning  Failed          8s (x2 over 24s)  kubelet  Error: SignatureValidationFailed
       

      Expected results:

          5 the image can be pulled successfully and the pod became running

      Additional info:

          On the pod's node, I can pull image successfully using podman or crictl. 
      sh-5.1# crictl pull quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
      Image is up to date for quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
      sh-5.1#     
      sh-5.1# crictl images | grep testpkisignedimage-1
      quay.io/minmli/testpkisignedimage-1              <none>              230662f80b00e       1.39MB
      
      sh-5.1# podman pull quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
      Trying to pull quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3...
      Getting image source signatures
      Checking if image destination supports signatures
      Copying blob f70adabe43c0 done   | 
      Copying config 230662f80b done   | 
      Writing manifest to image destination
      Storing signatures
      230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408
      
      
      The crio logs show: 
      Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302403078Z" level=info msg="Checking image status: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_status.go:31" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
      Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.30259278Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.skip_mount_home=true]quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\" does not resolve to an image ID" file="storage/storage_reference.go:154"
      Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302723181Z" level=debug msg="Can't find quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_status.go:126" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
      Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302769481Z" level=info msg="Image quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3 not found" file="server/image_status.go:145" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
      Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302860582Z" level=info msg="Neither image nor artfiact quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3 found" file="server/image_status.go:47" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
      ...
      Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.708198128Z" level=info msg="Pulling image: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_pull.go:44" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
      Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.708241229Z" level=debug msg="Using pull policy path for image quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3: \"\"" file="server/image_pull.go:147" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
      Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750278117Z" level=info msg="Trying to access \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\"" file="docker/docker_image_src.go:96"
      Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750438218Z" level=debug msg="Found credentials for quay.io/minmli/testpkisignedimage-1 in credential helper containers-auth.json in file /var/lib/kubelet/config.json" file="config/config.go:263"
      Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750478118Z" level=debug msg=" Lookaside configuration: using \"docker\" namespace quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="docker/registries_d.go:210"
      Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750513919Z" level=debug msg=" No signature storage configuration found for quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3, using built-in default file:///var/lib/containers/sigstore" file="docker/registries_d.go:183"
      Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750597619Z" level=debug msg=" Sigstore attachments: using \"docker\" namespace quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="docker/registries_d.go:243"
      Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.771852216Z" level=debug msg="GET https://quay.io/v2/auth?account=openshift-release-dev%2Bart_dev_priv_visitor_pull&scope=repository%3Aminmli%2Ftestpkisignedimage-1%3Apull&service=quay.io" file="docker/docker_client.go:845"
      Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.919422177Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/manifests/sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="docker/docker_client.go:616"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.462189484Z" level=debug msg="IsRunningImageAllowed for image docker:quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="signature/policy_eval.go:274"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.462468686Z" level=debug msg="Downloading /v2/minmli/testpkisignedimage-1/blobs/sha256:230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408" file="docker/docker_client.go:1051"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.462543987Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/blobs/sha256:230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408" file="docker/docker_client.go:616"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.598953145Z" level=debug msg="Reading /var/lib/containers/sigstore/minmli/testpkisignedimage-1@sha256=7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3/signature-1" file="docker/docker_image_src.go:543"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.599085247Z" level=debug msg="Looking for sigstore attachments in quay.io/minmli/testpkisignedimage-1:sha256-7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3.sig" file="docker/docker_client.go:1142"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.599138947Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/manifests/sha256-7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3.sig" file="docker/docker_client.go:616"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.678931183Z" level=debug msg="Downloading /v2/minmli/testpkisignedimage-1/blobs/sha256:7381d1e6c50aa97c5934e473e226db9a7c159754f7ad0d578f4e64c2852b395c" file="docker/docker_client.go:1051"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.678972183Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/blobs/sha256:7381d1e6c50aa97c5934e473e226db9a7c159754f7ad0d578f4e64c2852b395c" file="docker/docker_client.go:616"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.820416688Z" level=debug msg="Downloading /v2/minmli/testpkisignedimage-1/blobs/sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929" file="docker/docker_client.go:1051"
      Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.820485589Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/blobs/sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929" file="docker/docker_client.go:616"
      Dec 12 09:08:40 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:40.178773594Z" level=debug msg="added name \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\" to image \"230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408\"" file="storage/storage_dest.go:1624"
      Dec 12 09:08:40 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:40.179327099Z" level=info msg="Pulled image: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_pull.go:126" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
      Dec 12 09:08:40 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:40.1794169Z" level=debug msg="Response: *v1.PullImageResponse: image_ref:\"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\"" file="interceptors/interceptors.go:76" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
      ...
      Dec 12 09:08:52 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:52.317771557Z" level=debug msg="IsRunningImageAllowed for image docker:quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="signature/policy_eval.go:274"
      Dec 12 09:08:52 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:52.317809258Z" level=debug msg=" Using transport \"docker\" policy section \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\"" file="signature/policy_eval.go:143"
      Dec 12 09:08:52 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:52.319713675Z" level=debug msg="Response error: checking signature of \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\": verifying signatures: SignatureValidationFailed: Signature for identity \"quay.io/minmli/quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\" is not accepted" file="interceptors/interceptors.go:73" id=91708134-7f8b-4453-9f09-5e881b9fc1ca name=/runtime.v1.RuntimeService/CreateContainer
      

              qiwan233 Qi Wang
              rhn-support-minmli Min Li
              None
              None
              Min Li Min Li
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: