-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
4.21
-
None
-
None
-
False
-
-
None
-
Critical
-
Yes
-
None
-
None
-
Rejected
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Starting from 4.21 rc.0, ClusterImagePolicy is created by default with scope set to quay.io/openshift-release-dev/ocp-release. This caused restricted /etc/containers/policy.json to be created, requiring quay.io/openshift-release-dev/ocp-release image signature to be validated, even if IDMS is properly configured.
In airgapped cluster, fetching the image sig from quay fails, causing image pull to fail in cluster-version pod.
Version-Release number of selected component (if applicable):
4.21
How reproducible:
100% in disconnected (airgapped) env with 4.21 rc.0
Steps to Reproduce:
1. mirror quay.io 4.21 multi arch image payload to disconnected registry
oc adm release mirror -a /home/kni/combined-secret.json --keep-manifest-list=true --from="quay.io/openshift-release-dev/ocp-release:4.21.0-rc.1-multi" --to="registry.kni-qe-55.telco5gran.eng.rdu2.redhat.com:5000/openshift-release-dev/ocp-v4.0-art-dev" --to-release-image="registry.kni-qe-55.telco5gran.eng.rdu2.redhat.com:5000/openshift-release-dev/ocp-release"
2. set clusterimageset to quay.io image url in digest format (as recommended by assisted install doc, which avoids complications in IBI when 3 different disconnected registries involved).
3. Follow reference config in user git repo to install single IPv6 disconnected cluster with 4.21 rc.0 build.
4.
Actual results:
Installation fails due to cluster-version pod fails to pull quay.io ocp payload in digest format.
cluster installation fails with this error in cluster-version pod: message: 'Back-off pulling image "quay.io/openshift-release-dev/ocp-release@sha256:5c6e3e3e2a6ce82637ca851de628516d52481f5ce3b8f42982f1eb271f7a7164": SignatureValidationFailed: image pull failed for quay.io/openshift-release-dev/ocp-release@sha256:5c6e3e3e2a6ce82637ca851de628516d52481f5ce3b8f42982f1eb271f7a7164 because the signature validation failed: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists; artifact err: provided artifact is a container image'
reason: ImagePullBackOff
Expected results:
Installation succeeded
Additional info:
1. This issue currently does not manifest on X86 4.21 RC0 build (it only manifests in rc.0 multi-arch image used in ARM env), because the x86 release image url automatically changes to registry.ci.openshift.org image url for internal x86 build even though the quay.io image was specified in the clusterimageset. However, this issue can be manually reproduced by running podman pull any quay.io/openshift-release-dev/ocp-release from an airgapped x86 cluster, thus it will eventually manifest in production env even for x86 build as well. $ oc get pods -n openshift-cluster-version cluster-version-operator-b5fc6dbc8-zbsng -o yaml | grep -i image - --release-image=registry.ci.openshift.org/ocp/release@sha256:ecde621d6f74aa1af4cd351f8b571ca2a61bbc32826e49cdf1b7fbff07f04ede 2. This is not an issue in QE IPv4 cluster, the cluster was able to fetch the image sig from quay for the exact same image that failed on ARM ipv6, indicating the image sig exists. 3. This is not an issue in latest 4.20.z. No clusterimagepolicy was created by default, thus no image sig validation is required for any image. 4. Workaround via ZTP for 4.21: Add MC that is alphabetically after 99-master-generated-registries and 99-worker-generated-registries via extra manifests to override /etc/containers/policy.json config from the other two MCs. E.g., https://gitlab.cee.redhat.com/ocp-edge-qe/ztp-site-configs/-/blob/helix95-4.21/siteconfig/extra-manifests/99-zz-disable-image-sig-validation.yaml?ref_type=heads 5. Some log snippet: # auto created clusterimagepolicy in 4.21 rc.0: [kni@registry ~]$ oc get clusterimagepolicies.config.openshift.io openshift -o yaml apiVersion: config.openshift.io/v1 kind: ClusterImagePolicy metadata: annotations: exclude.release.openshift.io/internal-openshift-hosted: "true" include.release.openshift.io/self-managed-high-availability: "true" kubernetes.io/description: Require Red Hat signatures for quay.io/openshift-release-dev/ocp-release container images. creationTimestamp: "2025-12-29T21:11:35Z" generation: 1 name: openshift resourceVersion: "1115" uid: d3ef64a9-1899-4625-b54a-b139e8e693a4 spec: policy: rootOfTrust: policyType: PublicKey publicKey: keyData: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQ0lqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUEwQVN5dUgyVExXdkJVcVBIWjRJcAo3NWc3RW5jQmtnUUhkSm5qenhBVzVLUVRNaC9zaUJvQi9Cb1NydGlQTXduQ2hiVENuUU9JUWVadURpRm5odUo3Ck0vRDNiN0pvWDBtMTIzTmNDU242N21BZGpCYTZCZzZrdWtaZ0NQNFpVWmVFU2FqV1gvRWp5bEZjUkZPWFc1N3AKUkRDRU40MkovallsVnF0K2c5K0dya2VyOFN6ODZIM2wwdGJxT2RqYnovVnhIWWh3RjBjdFVNSHN5VlJEcTJRUAp0cXpOWGxtbE1oUy9Qb0ZyNlI0dS83SENuL0srTGVnY08yZkFGT2I0MEt2S1NLS1ZENmxld1VaRXJob3AxQ2dKClhqRHRHbW1POWRHTUY3MW1mNkhFZmFLU2R5K0VFNmlTRjJBMlZ2OVFoQmF3TWlxMmtPekVpTGc0bkFkSlQ4d2cKWnJNQW1QQ3FHSXNYTkdaNC9RK1lUd3dsY2UzZ2xxYjVMOXRmTm96RWRTUjlOODVERVNmUUxRRWRZM0NhbHdLTQpCVDFPRWhFWDF3SFJDVTRkck1PZWo2Qk5XMFZ0c2NHdEhtQ3JzNzRqUGV6aHdOVDh5cGt5UytUMHpUNFRzeTZmClZYa0o4WVNIeWVuU3pNQjJPcDJidnNFM2dyWStzNzRXaEc5VUlBNkRCeGNUaWUxNU5Tekt3Znphb05XT0RjTEYKcDdCWThhYUhFMk1xRnhZRlgrSWJqcGtRUmZhZVFRc291REZkQ2tYRUZWZlBwYkQyZGs2RmxlYU1UUHV5eHRJVApnalZFdEdRSzJxR0NGR2lRSEZkNGhmVitlQ0E2M0pybzF6MHpvQk01QmJJSVEzK2VWRnd0M0FsWnA1VVZ3cjZkCnNlY3FraS95cm12M1kwZHFaOVZPbjNVQ0F3RUFBUT09Ci0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQ== scopes: - quay.io/openshift-release-dev/ocp-release # auto generated registries MC now containers image sig validation info. [kni@registry ~]$ oc get mc 99-master-generated-registries -o yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: annotations: machineconfiguration.openshift.io/generated-by-controller-version: 4a8a503c3a969e948fb4dbeab19a5d3f0727d1c6 creationTimestamp: "2025-12-29T21:24:05Z" generation: 1 labels: machineconfiguration.openshift.io/role: master name: 99-master-generated-registries ... spec: baseOSExtensionsContainerImage: "" config: ignition: version: 3.5.0 storage: files: .... - contents: compression: "" source: data:text/plain;charset=utf-8;base64,ewogICJkZWZhdWx0IjogWwogICAgewogICAgICAidHlwZSI6ICJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIgogICAgfQogIF0sCiAgInRyYW5zcG9ydHMiOiB7CiAgICAiYXRvbWljIjogewogICAgICAicXVheS5pby9vcGVuc2hpZnQtcmVsZWFzZS1kZXYvb2NwLXJlbGVhc2UiOiBbCiAgICAgICAgewogICAgICAgICAgInR5cGUiOiAic2lnc3RvcmVTaWduZWQiLAogICAgICAgICAgImtleURhdGEiOiAiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVsSlEwbHFRVTVDWjJ0eGFHdHBSemwzTUVKQlVVVkdRVUZQUTBGbk9FRk5TVWxEUTJkTFEwRm5SVUV3UVZONWRVZ3lWRXhYZGtKVmNWQklXalJKY0FvM05XYzNSVzVqUW10blVVaGtTbTVxZW5oQlZ6VkxVVlJOYUM5emFVSnZRaTlDYjFOeWRHbFFUWGR1UTJoaVZFTnVVVTlKVVdWYWRVUnBSbTVvZFVvM0NrMHZSRE5pTjBwdldEQnRNVEl6VG1ORFUyNDJOMjFCWkdwQ1lUWkNaelpyZFd0YVowTlFORnBWV21WRlUyRnFWMWd2UldwNWJFWmpVa1pQV0ZjMU4zQUtVa1JEUlU0ME1rb3ZhbGxzVm5GMEsyYzVLMGR5YTJWeU9GTjZPRFpJTTJ3d2RHSnhUMlJxWW5vdlZuaElXV2gzUmpCamRGVk5TSE41VmxKRWNUSlJVQXAwY1hwT1dHeHRiRTFvVXk5UWIwWnlObEkwZFM4M1NFTnVMMHNyVEdWblkwOHlaa0ZHVDJJME1FdDJTMU5MUzFaRU5teGxkMVZhUlhKb2IzQXhRMmRLQ2xocVJIUkhiVzFQT1dSSFRVWTNNVzFtTmtoRlptRkxVMlI1SzBWRk5tbFRSakpCTWxaMk9WRm9RbUYzVFdseE1tdFBla1ZwVEdjMGJrRmtTbFE0ZDJjS1duSk5RVzFRUTNGSFNYTllUa2RhTkM5UksxbFVkM2RzWTJVeloyeHhZalZNT1hSbVRtOTZSV1JUVWpsT09EVkVSVk5tVVV4UlJXUlpNME5oYkhkTFRRcENWREZQUldoRldERjNTRkpEVlRSa2NrMVBaV28yUWs1WE1GWjBjMk5IZEVodFEzSnpOelJxVUdWNmFIZE9WRGg1Y0d0NVV5dFVNSHBVTkZSemVUWm1DbFpZYTBvNFdWTkllV1Z1VTNwTlFqSlBjREppZG5ORk0yZHlXU3R6TnpSWGFFYzVWVWxCTmtSQ2VHTlVhV1V4TlU1VGVrdDNabnBoYjA1WFQwUmpURVlLY0RkQ1dUaGhZVWhGTWsxeFJuaFpSbGdyU1dKcWNHdFJVbVpoWlZGUmMyOTFSRVprUTJ0WVJVWldabEJ3WWtReVpHczJSbXhsWVUxVVVIVjVlSFJKVkFwbmFsWkZkRWRSU3pKeFIwTkdSMmxSU0Vaa05HaG1WaXRsUTBFMk0wcHliekY2TUhwdlFrMDFRbUpKU1ZFeksyVldSbmQwTTBGc1duQTFWVlozY2paa0NuTmxZM0ZyYVM5NWNtMTJNMWt3WkhGYU9WWlBiak5WUTBGM1JVRkJVVDA5Q2kwdExTMHRSVTVFSUZCVlFreEpReUJMUlZrdExTMHRMUT09IiwKICAgICAgICAgICJzaWduZWRJZGVudGl0eSI6IHsKICAgICAgICAgICAgInR5cGUiOiAibWF0Y2hSZXBvRGlnZXN0T3JFeGFjdCIKICAgICAgICAgIH0KICAgICAgICB9CiAgICAgIF0KICAgIH0sCiAgICAiZG9ja2VyIjogewogICAgICAicXVheS5pby9vcGVuc2hpZnQtcmVsZWFzZS1kZXYvb2NwLXJlbGVhc2UiOiBbCiAgICAgICAgewogICAgICAgICAgInR5cGUiOiAic2lnc3RvcmVTaWduZWQiLAogICAgICAgICAgImtleURhdGEiOiAiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVsSlEwbHFRVTVDWjJ0eGFHdHBSemwzTUVKQlVVVkdRVUZQUTBGbk9FRk5TVWxEUTJkTFEwRm5SVUV3UVZONWRVZ3lWRXhYZGtKVmNWQklXalJKY0FvM05XYzNSVzVqUW10blVVaGtTbTVxZW5oQlZ6VkxVVlJOYUM5emFVSnZRaTlDYjFOeWRHbFFUWGR1UTJoaVZFTnVVVTlKVVdWYWRVUnBSbTVvZFVvM0NrMHZSRE5pTjBwdldEQnRNVEl6VG1ORFUyNDJOMjFCWkdwQ1lUWkNaelpyZFd0YVowTlFORnBWV21WRlUyRnFWMWd2UldwNWJFWmpVa1pQV0ZjMU4zQUtVa1JEUlU0ME1rb3ZhbGxzVm5GMEsyYzVLMGR5YTJWeU9GTjZPRFpJTTJ3d2RHSnhUMlJxWW5vdlZuaElXV2gzUmpCamRGVk5TSE41VmxKRWNUSlJVQXAwY1hwT1dHeHRiRTFvVXk5UWIwWnlObEkwZFM4M1NFTnVMMHNyVEdWblkwOHlaa0ZHVDJJME1FdDJTMU5MUzFaRU5teGxkMVZhUlhKb2IzQXhRMmRLQ2xocVJIUkhiVzFQT1dSSFRVWTNNVzFtTmtoRlptRkxVMlI1SzBWRk5tbFRSakpCTWxaMk9WRm9RbUYzVFdseE1tdFBla1ZwVEdjMGJrRmtTbFE0ZDJjS1duSk5RVzFRUTNGSFNYTllUa2RhTkM5UksxbFVkM2RzWTJVeloyeHhZalZNT1hSbVRtOTZSV1JUVWpsT09EVkVSVk5tVVV4UlJXUlpNME5oYkhkTFRRcENWREZQUldoRldERjNTRkpEVlRSa2NrMVBaV28yUWs1WE1GWjBjMk5IZEVodFEzSnpOelJxVUdWNmFIZE9WRGg1Y0d0NVV5dFVNSHBVTkZSemVUWm1DbFpZYTBvNFdWTkllV1Z1VTNwTlFqSlBjREppZG5ORk0yZHlXU3R6TnpSWGFFYzVWVWxCTmtSQ2VHTlVhV1V4TlU1VGVrdDNabnBoYjA1WFQwUmpURVlLY0RkQ1dUaGhZVWhGTWsxeFJuaFpSbGdyU1dKcWNHdFJVbVpoWlZGUmMyOTFSRVprUTJ0WVJVWldabEJ3WWtReVpHczJSbXhsWVUxVVVIVjVlSFJKVkFwbmFsWkZkRWRSU3pKeFIwTkdSMmxSU0Vaa05HaG1WaXRsUTBFMk0wcHliekY2TUhwdlFrMDFRbUpKU1ZFeksyVldSbmQwTTBGc1duQTFWVlozY2paa0NuTmxZM0ZyYVM5NWNtMTJNMWt3WkhGYU9WWlBiak5WUTBGM1JVRkJVVDA5Q2kwdExTMHRSVTVFSUZCVlFreEpReUJMUlZrdExTMHRMUT09IiwKICAgICAgICAgICJzaWduZWRJZGVudGl0eSI6IHsKICAgICAgICAgICAgInR5cGUiOiAibWF0Y2hSZXBvRGlnZXN0T3JFeGFjdCIKICAgICAgICAgIH0KICAgICAgICB9CiAgICAgIF0KICAgIH0sCiAgICAiZG9ja2VyLWRhZW1vbiI6IHsKICAgICAgIiI6IFsKICAgICAgICB7CiAgICAgICAgICAidHlwZSI6ICJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIgogICAgICAgIH0KICAgICAgXQogICAgfQogIH0KfQ== mode: 420 overwrite: true path: /etc/containers/policy.json - contents: compression: "" source: data:text/plain;charset=utf-8;base64,ZG9ja2VyOgogIHF1YXkuaW8vb3BlbnNoaWZ0LXJlbGVhc2UtZGV2L29jcC1yZWxlYXNlOgogICAgdXNlLXNpZ3N0b3JlLWF0dGFjaG1lbnRzOiB0cnVlCiAgcmVnaXN0cnkua25pLXFlLTU1LnRlbGNvNWdyYW4uZW5nLnJkdTIucmVkaGF0LmNvbTo1MDAwL29wZW5zaGlmdC1yZWxlYXNlLWRldi9vY3AtcmVsZWFzZToKICAgIHVzZS1zaWdzdG9yZS1hdHRhY2htZW50czogdHJ1ZQogIHJlZ2lzdHJ5LmtuaS1xZS01NS50ZWxjbzVncmFuLmVuZy5yZHUyLnJlZGhhdC5jb206NTAwMC9vcGVuc2hpZnQtcmVsZWFzZS1kZXYvb2NwLXY0LjAtYXJ0LWRldjoKICAgIHVzZS1zaWdzdG9yZS1hdHRhY2htZW50czogdHJ1ZQo= mode: 420 overwrite: true path: /etc/containers/registries.d/sigstore-registries.yaml fips: false kernelArguments: null kernelType: "" osImageURL: ""
- is caused by
-
OCPSTRAT-2471 Enable sigstore 'openshift' clusterimagepolicy by default to enable payload index image verification
-
- Release Pending
-