-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
4.21
-
None
-
None
-
False
-
-
None
-
Important
-
None
-
None
-
None
-
Rejected
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
SignatureValidationFailed when creating a pod after applying an image policy with PKI. The image used is signed with cosign PKI. (refer to https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-86101)
Version-Release number of selected component (if applicable):
4.21.0-0.nightly-2025-12-10-193741
How reproducible:
Always
Steps to Reproduce:
1. Set crio log level as "debug"
#oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
name: set-loglevel
spec:
machineConfigPoolSelector:
matchLabels:
pools.operator.machineconfiguration.openshift.io/worker: ""
containerRuntimeConfig:
logLevel: debug
EOF
2. create a user namespace
% oc new-project testnamespace
3. create an imagepolicy crd
apiVersion: config.openshift.io/v1
kind: ImagePolicy
metadata:
name: byopkipolicy # BYOPKI with Root CA and Intermediate CA
namespace: testnamespace
spec:
scopes:
- quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
policy:
rootOfTrust:
policyType: PKI
pki:
caRootsData: LS0tLS1CRUdJT....Q0FURS0tLS0t
caIntermediatesData: LS0tLS1CRUd....JQ0FURS0tLS0t
pkiCertificateSubject:
email: minmli@redhat.com
hostname: myhost.example.com
4. wait a few minutes, check the policy take effect on worker
sh-5.1# cat /etc/crio/policies/testnamespace.json
sh-5.1# cat /etc/containers/registries.d/sigstore-registries.yaml
docker:
quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3:
use-sigstore-attachments: true
5. create a pod with the signed image
#oc apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
name: testpod-1
labels:
app: testpod
spec:
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: hello-pod
image: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
ports:
- containerPort: 80
EOF
Actual results:
5 failed to pull image due to error : SignatureValidationFailed
% oc get pod
NAME READY STATUS RESTARTS AGE
testpod-1 0/1 SignatureValidationFailed 0 5s
% oc describe pod testpod-1
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedInterface 24s multus Add eth0 [10.128.2.22/23] from ovn-kubernetes
Normal BackOff 23s kubelet Back-off pulling image "quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3"
Warning Failed 23s kubelet Error: ImagePullBackOff
Normal Pulling 9s (x2 over 24s) kubelet Pulling image "quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3"
Warning Failed 8s (x2 over 24s) kubelet Failed to pull image "quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3": SignatureValidationFailed: unable to pull image or OCI artifact: pull image err: Source image rejected: Signature for identity "quay.io/minmli/quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" is not accepted; artifact err: provided artifact is a container image
Warning Failed 8s (x2 over 24s) kubelet Error: SignatureValidationFailed
Expected results:
5 the image can be pulled successfully and the pod became running
Additional info:
On the pod's node, I can pull image successfully using podman or crictl.
sh-5.1# crictl pull quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
Image is up to date for quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
sh-5.1#
sh-5.1# crictl images | grep testpkisignedimage-1
quay.io/minmli/testpkisignedimage-1 <none> 230662f80b00e 1.39MB
sh-5.1# podman pull quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3
Trying to pull quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3...
Getting image source signatures
Checking if image destination supports signatures
Copying blob f70adabe43c0 done |
Copying config 230662f80b done |
Writing manifest to image destination
Storing signatures
230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408
The crio logs show:
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302403078Z" level=info msg="Checking image status: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_status.go:31" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.30259278Z" level=debug msg="reference \"[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.skip_mount_home=true]quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\" does not resolve to an image ID" file="storage/storage_reference.go:154"
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302723181Z" level=debug msg="Can't find quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_status.go:126" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302769481Z" level=info msg="Image quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3 not found" file="server/image_status.go:145" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
Dec 12 09:04:28 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:04:28.302860582Z" level=info msg="Neither image nor artfiact quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3 found" file="server/image_status.go:47" id=2dca236a-6acc-497f-b2f3-16f5a20e2496 name=/runtime.v1.ImageService/ImageStatus
...
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.708198128Z" level=info msg="Pulling image: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_pull.go:44" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.708241229Z" level=debug msg="Using pull policy path for image quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3: \"\"" file="server/image_pull.go:147" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750278117Z" level=info msg="Trying to access \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\"" file="docker/docker_image_src.go:96"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750438218Z" level=debug msg="Found credentials for quay.io/minmli/testpkisignedimage-1 in credential helper containers-auth.json in file /var/lib/kubelet/config.json" file="config/config.go:263"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750478118Z" level=debug msg=" Lookaside configuration: using \"docker\" namespace quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="docker/registries_d.go:210"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750513919Z" level=debug msg=" No signature storage configuration found for quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3, using built-in default file:///var/lib/containers/sigstore" file="docker/registries_d.go:183"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.750597619Z" level=debug msg=" Sigstore attachments: using \"docker\" namespace quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="docker/registries_d.go:243"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.771852216Z" level=debug msg="GET https://quay.io/v2/auth?account=openshift-release-dev%2Bart_dev_priv_visitor_pull&scope=repository%3Aminmli%2Ftestpkisignedimage-1%3Apull&service=quay.io" file="docker/docker_client.go:845"
Dec 12 09:08:38 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:38.919422177Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/manifests/sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="docker/docker_client.go:616"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.462189484Z" level=debug msg="IsRunningImageAllowed for image docker:quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="signature/policy_eval.go:274"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.462468686Z" level=debug msg="Downloading /v2/minmli/testpkisignedimage-1/blobs/sha256:230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408" file="docker/docker_client.go:1051"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.462543987Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/blobs/sha256:230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408" file="docker/docker_client.go:616"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.598953145Z" level=debug msg="Reading /var/lib/containers/sigstore/minmli/testpkisignedimage-1@sha256=7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3/signature-1" file="docker/docker_image_src.go:543"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.599085247Z" level=debug msg="Looking for sigstore attachments in quay.io/minmli/testpkisignedimage-1:sha256-7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3.sig" file="docker/docker_client.go:1142"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.599138947Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/manifests/sha256-7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3.sig" file="docker/docker_client.go:616"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.678931183Z" level=debug msg="Downloading /v2/minmli/testpkisignedimage-1/blobs/sha256:7381d1e6c50aa97c5934e473e226db9a7c159754f7ad0d578f4e64c2852b395c" file="docker/docker_client.go:1051"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.678972183Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/blobs/sha256:7381d1e6c50aa97c5934e473e226db9a7c159754f7ad0d578f4e64c2852b395c" file="docker/docker_client.go:616"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.820416688Z" level=debug msg="Downloading /v2/minmli/testpkisignedimage-1/blobs/sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929" file="docker/docker_client.go:1051"
Dec 12 09:08:39 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:39.820485589Z" level=debug msg="GET https://quay.io/v2/minmli/testpkisignedimage-1/blobs/sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929" file="docker/docker_client.go:616"
Dec 12 09:08:40 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:40.178773594Z" level=debug msg="added name \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\" to image \"230662f80b00efe44d7f65d4d111f5fe9914f3dad02e92a9120995059f860408\"" file="storage/storage_dest.go:1624"
Dec 12 09:08:40 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:40.179327099Z" level=info msg="Pulled image: quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="server/image_pull.go:126" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
Dec 12 09:08:40 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:40.1794169Z" level=debug msg="Response: *v1.PullImageResponse: image_ref:\"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\"" file="interceptors/interceptors.go:76" id=5f889081-5de4-4b13-b48e-1a4a2e2540ce name=/runtime.v1.ImageService/PullImage
...
Dec 12 09:08:52 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:52.317771557Z" level=debug msg="IsRunningImageAllowed for image docker:quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3" file="signature/policy_eval.go:274"
Dec 12 09:08:52 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:52.317809258Z" level=debug msg=" Using transport \"docker\" policy section \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\"" file="signature/policy_eval.go:143"
Dec 12 09:08:52 minmli-121201-grdms-worker-eastus-1 crio[2538]: time="2025-12-12T09:08:52.319713675Z" level=debug msg="Response error: checking signature of \"quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\": verifying signatures: SignatureValidationFailed: Signature for identity \"quay.io/minmli/quay.io/minmli/testpkisignedimage-1@sha256:7fe37303835533b708fd69c8a4aa1b48b532917561ad545c290c9297791338e3\" is not accepted" file="interceptors/interceptors.go:73" id=91708134-7f8b-4453-9f09-5e881b9fc1ca name=/runtime.v1.RuntimeService/CreateContainer