Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-70261

Release payload image pull fails in disconnected env with ClusterImagePolicy feature enabled by default in 4.21

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.21
    • GitOps ZTP
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Critical
    • Yes
    • None
    • None
    • Rejected
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          Starting from 4.21 rc.0, ClusterImagePolicy is created by default with scope set to quay.io/openshift-release-dev/ocp-release. This caused restricted /etc/containers/policy.json to be created, requiring quay.io/openshift-release-dev/ocp-release image signature to be validated, even if IDMS is properly configured. 
      
      In airgapped cluster, fetching the image sig from quay fails, causing image pull to fail in cluster-version pod. 

      Version-Release number of selected component (if applicable):

          4.21

      How reproducible:

          100% in disconnected (airgapped) env with 4.21 rc.0 

      Steps to Reproduce:

          1. mirror quay.io 4.21 multi arch image payload to disconnected registry
      
      oc adm release mirror -a /home/kni/combined-secret.json  --keep-manifest-list=true  --from="quay.io/openshift-release-dev/ocp-release:4.21.0-rc.1-multi"  --to="registry.kni-qe-55.telco5gran.eng.rdu2.redhat.com:5000/openshift-release-dev/ocp-v4.0-art-dev"  --to-release-image="registry.kni-qe-55.telco5gran.eng.rdu2.redhat.com:5000/openshift-release-dev/ocp-release"
      
           2. set clusterimageset to quay.io image url in digest format (as recommended by assisted install doc, which avoids complications in IBI when 3 different disconnected registries involved).
          3. Follow reference config in user git repo to install single IPv6 disconnected cluster with 4.21 rc.0 build.      
          4.
          

      Actual results:

          Installation fails due to cluster-version pod fails to pull quay.io ocp payload in digest format. 
      
       cluster installation fails with this error in  cluster-version pod:       message: 'Back-off pulling image "quay.io/openshift-release-dev/ocp-release@sha256:5c6e3e3e2a6ce82637ca851de628516d52481f5ce3b8f42982f1eb271f7a7164":          SignatureValidationFailed: image pull failed for quay.io/openshift-release-dev/ocp-release@sha256:5c6e3e3e2a6ce82637ca851de628516d52481f5ce3b8f42982f1eb271f7a7164          because the signature validation failed: unable to pull image or OCI artifact:          pull image err: copying system image from manifest list: Source image rejected:          A signature was required, but no signature exists; artifact err: provided          artifact is a container image'        
      reason: ImagePullBackOff

      Expected results:

          Installation succeeded

      Additional info:

      1. This issue currently does not manifest on X86 4.21 RC0 build (it only manifests in rc.0 multi-arch image used in ARM env), because the x86 release image url automatically changes to registry.ci.openshift.org image url for internal x86 build even though the quay.io image was specified in the clusterimageset. However, this issue can be manually reproduced by running podman pull any quay.io/openshift-release-dev/ocp-release from an airgapped x86 cluster, thus it will eventually manifest in production env even for x86 build as well.
       
      $ oc get pods -n openshift-cluster-version cluster-version-operator-b5fc6dbc8-zbsng -o yaml | grep -i image
          - --release-image=registry.ci.openshift.org/ocp/release@sha256:ecde621d6f74aa1af4cd351f8b571ca2a61bbc32826e49cdf1b7fbff07f04ede
      
      
      2. This is not an issue in QE IPv4 cluster, the cluster was able to fetch the image sig from quay for the exact same image that failed on ARM ipv6, indicating the image sig exists. 
      
      
      3. This is not an issue in latest 4.20.z. No clusterimagepolicy was created by default, thus no image sig validation is required for any image.
      
      
      4. Workaround via ZTP for 4.21: 
      
      Add MC that is alphabetically after 99-master-generated-registries and 99-worker-generated-registries via extra manifests to override /etc/containers/policy.json config from the other two MCs. 
      
      E.g.,  https://gitlab.cee.redhat.com/ocp-edge-qe/ztp-site-configs/-/blob/helix95-4.21/siteconfig/extra-manifests/99-zz-disable-image-sig-validation.yaml?ref_type=heads 
      
      
      5. Some log snippet: 
      
      # auto created clusterimagepolicy in 4.21 rc.0:
      [kni@registry ~]$ oc get clusterimagepolicies.config.openshift.io openshift -o yaml 
      apiVersion: config.openshift.io/v1
      kind: ClusterImagePolicy
      metadata:
        annotations:
          exclude.release.openshift.io/internal-openshift-hosted: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          kubernetes.io/description: Require Red Hat signatures for quay.io/openshift-release-dev/ocp-release
            container images.
        creationTimestamp: "2025-12-29T21:11:35Z"
        generation: 1
        name: openshift
        resourceVersion: "1115"
        uid: d3ef64a9-1899-4625-b54a-b139e8e693a4
      spec:
        policy:
          rootOfTrust:
            policyType: PublicKey
            publicKey:
              keyData: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQ0lqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FnOEFNSUlDQ2dLQ0FnRUEwQVN5dUgyVExXdkJVcVBIWjRJcAo3NWc3RW5jQmtnUUhkSm5qenhBVzVLUVRNaC9zaUJvQi9Cb1NydGlQTXduQ2hiVENuUU9JUWVadURpRm5odUo3Ck0vRDNiN0pvWDBtMTIzTmNDU242N21BZGpCYTZCZzZrdWtaZ0NQNFpVWmVFU2FqV1gvRWp5bEZjUkZPWFc1N3AKUkRDRU40MkovallsVnF0K2c5K0dya2VyOFN6ODZIM2wwdGJxT2RqYnovVnhIWWh3RjBjdFVNSHN5VlJEcTJRUAp0cXpOWGxtbE1oUy9Qb0ZyNlI0dS83SENuL0srTGVnY08yZkFGT2I0MEt2S1NLS1ZENmxld1VaRXJob3AxQ2dKClhqRHRHbW1POWRHTUY3MW1mNkhFZmFLU2R5K0VFNmlTRjJBMlZ2OVFoQmF3TWlxMmtPekVpTGc0bkFkSlQ4d2cKWnJNQW1QQ3FHSXNYTkdaNC9RK1lUd3dsY2UzZ2xxYjVMOXRmTm96RWRTUjlOODVERVNmUUxRRWRZM0NhbHdLTQpCVDFPRWhFWDF3SFJDVTRkck1PZWo2Qk5XMFZ0c2NHdEhtQ3JzNzRqUGV6aHdOVDh5cGt5UytUMHpUNFRzeTZmClZYa0o4WVNIeWVuU3pNQjJPcDJidnNFM2dyWStzNzRXaEc5VUlBNkRCeGNUaWUxNU5Tekt3Znphb05XT0RjTEYKcDdCWThhYUhFMk1xRnhZRlgrSWJqcGtRUmZhZVFRc291REZkQ2tYRUZWZlBwYkQyZGs2RmxlYU1UUHV5eHRJVApnalZFdEdRSzJxR0NGR2lRSEZkNGhmVitlQ0E2M0pybzF6MHpvQk01QmJJSVEzK2VWRnd0M0FsWnA1VVZ3cjZkCnNlY3FraS95cm12M1kwZHFaOVZPbjNVQ0F3RUFBUT09Ci0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQ==
        scopes:
        - quay.io/openshift-release-dev/ocp-release
      
      # auto generated registries MC now containers image sig validation info.
      [kni@registry ~]$ oc get mc 99-master-generated-registries  -o yaml
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        annotations:
          machineconfiguration.openshift.io/generated-by-controller-version: 4a8a503c3a969e948fb4dbeab19a5d3f0727d1c6
        creationTimestamp: "2025-12-29T21:24:05Z"
        generation: 1
        labels:
          machineconfiguration.openshift.io/role: master
        name: 99-master-generated-registries
      ...
      spec:
        baseOSExtensionsContainerImage: ""
        config:
          ignition:
            version: 3.5.0
          storage:
            files:
      ....
            - contents:
                compression: ""
                source: data:text/plain;charset=utf-8;base64,ewogICJkZWZhdWx0IjogWwogICAgewogICAgICAidHlwZSI6ICJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIgogICAgfQogIF0sCiAgInRyYW5zcG9ydHMiOiB7CiAgICAiYXRvbWljIjogewogICAgICAicXVheS5pby9vcGVuc2hpZnQtcmVsZWFzZS1kZXYvb2NwLXJlbGVhc2UiOiBbCiAgICAgICAgewogICAgICAgICAgInR5cGUiOiAic2lnc3RvcmVTaWduZWQiLAogICAgICAgICAgImtleURhdGEiOiAiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVsSlEwbHFRVTVDWjJ0eGFHdHBSemwzTUVKQlVVVkdRVUZQUTBGbk9FRk5TVWxEUTJkTFEwRm5SVUV3UVZONWRVZ3lWRXhYZGtKVmNWQklXalJKY0FvM05XYzNSVzVqUW10blVVaGtTbTVxZW5oQlZ6VkxVVlJOYUM5emFVSnZRaTlDYjFOeWRHbFFUWGR1UTJoaVZFTnVVVTlKVVdWYWRVUnBSbTVvZFVvM0NrMHZSRE5pTjBwdldEQnRNVEl6VG1ORFUyNDJOMjFCWkdwQ1lUWkNaelpyZFd0YVowTlFORnBWV21WRlUyRnFWMWd2UldwNWJFWmpVa1pQV0ZjMU4zQUtVa1JEUlU0ME1rb3ZhbGxzVm5GMEsyYzVLMGR5YTJWeU9GTjZPRFpJTTJ3d2RHSnhUMlJxWW5vdlZuaElXV2gzUmpCamRGVk5TSE41VmxKRWNUSlJVQXAwY1hwT1dHeHRiRTFvVXk5UWIwWnlObEkwZFM4M1NFTnVMMHNyVEdWblkwOHlaa0ZHVDJJME1FdDJTMU5MUzFaRU5teGxkMVZhUlhKb2IzQXhRMmRLQ2xocVJIUkhiVzFQT1dSSFRVWTNNVzFtTmtoRlptRkxVMlI1SzBWRk5tbFRSakpCTWxaMk9WRm9RbUYzVFdseE1tdFBla1ZwVEdjMGJrRmtTbFE0ZDJjS1duSk5RVzFRUTNGSFNYTllUa2RhTkM5UksxbFVkM2RzWTJVeloyeHhZalZNT1hSbVRtOTZSV1JUVWpsT09EVkVSVk5tVVV4UlJXUlpNME5oYkhkTFRRcENWREZQUldoRldERjNTRkpEVlRSa2NrMVBaV28yUWs1WE1GWjBjMk5IZEVodFEzSnpOelJxVUdWNmFIZE9WRGg1Y0d0NVV5dFVNSHBVTkZSemVUWm1DbFpZYTBvNFdWTkllV1Z1VTNwTlFqSlBjREppZG5ORk0yZHlXU3R6TnpSWGFFYzVWVWxCTmtSQ2VHTlVhV1V4TlU1VGVrdDNabnBoYjA1WFQwUmpURVlLY0RkQ1dUaGhZVWhGTWsxeFJuaFpSbGdyU1dKcWNHdFJVbVpoWlZGUmMyOTFSRVprUTJ0WVJVWldabEJ3WWtReVpHczJSbXhsWVUxVVVIVjVlSFJKVkFwbmFsWkZkRWRSU3pKeFIwTkdSMmxSU0Vaa05HaG1WaXRsUTBFMk0wcHliekY2TUhwdlFrMDFRbUpKU1ZFeksyVldSbmQwTTBGc1duQTFWVlozY2paa0NuTmxZM0ZyYVM5NWNtMTJNMWt3WkhGYU9WWlBiak5WUTBGM1JVRkJVVDA5Q2kwdExTMHRSVTVFSUZCVlFreEpReUJMUlZrdExTMHRMUT09IiwKICAgICAgICAgICJzaWduZWRJZGVudGl0eSI6IHsKICAgICAgICAgICAgInR5cGUiOiAibWF0Y2hSZXBvRGlnZXN0T3JFeGFjdCIKICAgICAgICAgIH0KICAgICAgICB9CiAgICAgIF0KICAgIH0sCiAgICAiZG9ja2VyIjogewogICAgICAicXVheS5pby9vcGVuc2hpZnQtcmVsZWFzZS1kZXYvb2NwLXJlbGVhc2UiOiBbCiAgICAgICAgewogICAgICAgICAgInR5cGUiOiAic2lnc3RvcmVTaWduZWQiLAogICAgICAgICAgImtleURhdGEiOiAiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVsSlEwbHFRVTVDWjJ0eGFHdHBSemwzTUVKQlVVVkdRVUZQUTBGbk9FRk5TVWxEUTJkTFEwRm5SVUV3UVZONWRVZ3lWRXhYZGtKVmNWQklXalJKY0FvM05XYzNSVzVqUW10blVVaGtTbTVxZW5oQlZ6VkxVVlJOYUM5emFVSnZRaTlDYjFOeWRHbFFUWGR1UTJoaVZFTnVVVTlKVVdWYWRVUnBSbTVvZFVvM0NrMHZSRE5pTjBwdldEQnRNVEl6VG1ORFUyNDJOMjFCWkdwQ1lUWkNaelpyZFd0YVowTlFORnBWV21WRlUyRnFWMWd2UldwNWJFWmpVa1pQV0ZjMU4zQUtVa1JEUlU0ME1rb3ZhbGxzVm5GMEsyYzVLMGR5YTJWeU9GTjZPRFpJTTJ3d2RHSnhUMlJxWW5vdlZuaElXV2gzUmpCamRGVk5TSE41VmxKRWNUSlJVQXAwY1hwT1dHeHRiRTFvVXk5UWIwWnlObEkwZFM4M1NFTnVMMHNyVEdWblkwOHlaa0ZHVDJJME1FdDJTMU5MUzFaRU5teGxkMVZhUlhKb2IzQXhRMmRLQ2xocVJIUkhiVzFQT1dSSFRVWTNNVzFtTmtoRlptRkxVMlI1SzBWRk5tbFRSakpCTWxaMk9WRm9RbUYzVFdseE1tdFBla1ZwVEdjMGJrRmtTbFE0ZDJjS1duSk5RVzFRUTNGSFNYTllUa2RhTkM5UksxbFVkM2RzWTJVeloyeHhZalZNT1hSbVRtOTZSV1JUVWpsT09EVkVSVk5tVVV4UlJXUlpNME5oYkhkTFRRcENWREZQUldoRldERjNTRkpEVlRSa2NrMVBaV28yUWs1WE1GWjBjMk5IZEVodFEzSnpOelJxVUdWNmFIZE9WRGg1Y0d0NVV5dFVNSHBVTkZSemVUWm1DbFpZYTBvNFdWTkllV1Z1VTNwTlFqSlBjREppZG5ORk0yZHlXU3R6TnpSWGFFYzVWVWxCTmtSQ2VHTlVhV1V4TlU1VGVrdDNabnBoYjA1WFQwUmpURVlLY0RkQ1dUaGhZVWhGTWsxeFJuaFpSbGdyU1dKcWNHdFJVbVpoWlZGUmMyOTFSRVprUTJ0WVJVWldabEJ3WWtReVpHczJSbXhsWVUxVVVIVjVlSFJKVkFwbmFsWkZkRWRSU3pKeFIwTkdSMmxSU0Vaa05HaG1WaXRsUTBFMk0wcHliekY2TUhwdlFrMDFRbUpKU1ZFeksyVldSbmQwTTBGc1duQTFWVlozY2paa0NuTmxZM0ZyYVM5NWNtMTJNMWt3WkhGYU9WWlBiak5WUTBGM1JVRkJVVDA5Q2kwdExTMHRSVTVFSUZCVlFreEpReUJMUlZrdExTMHRMUT09IiwKICAgICAgICAgICJzaWduZWRJZGVudGl0eSI6IHsKICAgICAgICAgICAgInR5cGUiOiAibWF0Y2hSZXBvRGlnZXN0T3JFeGFjdCIKICAgICAgICAgIH0KICAgICAgICB9CiAgICAgIF0KICAgIH0sCiAgICAiZG9ja2VyLWRhZW1vbiI6IHsKICAgICAgIiI6IFsKICAgICAgICB7CiAgICAgICAgICAidHlwZSI6ICJpbnNlY3VyZUFjY2VwdEFueXRoaW5nIgogICAgICAgIH0KICAgICAgXQogICAgfQogIH0KfQ==
              mode: 420
              overwrite: true
              path: /etc/containers/policy.json
            - contents:
                compression: ""
                source: data:text/plain;charset=utf-8;base64,ZG9ja2VyOgogIHF1YXkuaW8vb3BlbnNoaWZ0LXJlbGVhc2UtZGV2L29jcC1yZWxlYXNlOgogICAgdXNlLXNpZ3N0b3JlLWF0dGFjaG1lbnRzOiB0cnVlCiAgcmVnaXN0cnkua25pLXFlLTU1LnRlbGNvNWdyYW4uZW5nLnJkdTIucmVkaGF0LmNvbTo1MDAwL29wZW5zaGlmdC1yZWxlYXNlLWRldi9vY3AtcmVsZWFzZToKICAgIHVzZS1zaWdzdG9yZS1hdHRhY2htZW50czogdHJ1ZQogIHJlZ2lzdHJ5LmtuaS1xZS01NS50ZWxjbzVncmFuLmVuZy5yZHUyLnJlZGhhdC5jb206NTAwMC9vcGVuc2hpZnQtcmVsZWFzZS1kZXYvb2NwLXY0LjAtYXJ0LWRldjoKICAgIHVzZS1zaWdzdG9yZS1hdHRhY2htZW50czogdHJ1ZQo=
              mode: 420
              overwrite: true
              path: /etc/containers/registries.d/sigstore-registries.yaml
        fips: false
        kernelArguments: null
        kernelType: ""
        osImageURL: ""
      
      
      

              rh-ee-apalanis Abraham Miller
              rhn-support-yliu1 Yang Liu
              None
              Irina Mihai
              Joshua Clark Joshua Clark
              None
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated: