Customer is installing a cluster via ACM in disconnected environment. Surprisingly, some of the OCP components are pulling the images with sha values different than what we see in the mirrorring logs.

When checked further, found that the images are the same but one of the sha value is not available on mirroring and that is what the components try to pull.

For example, cloud-credential-operator tries to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5eec5d1cde5c71750b9b4412411b4b747fb80bed05663b8b371a0ab970663cd8 but it didn't get mirrored. The podman inspect of this image shows below repo digests.
~~~
          "RepoDigests": [
               "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b514efab0b38704677ef213a991960df181264c480ec7886bf251b6f2a81fce",
               "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5eec5d1cde5c71750b9b4412411b4b747fb80bed05663b8b371a0ab970663cd8"
~~~

Looking at https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/4.14.18/release.txt, it is seen that quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b514efab0b38704677ef213a991960df181264c480ec7886bf251b6f2a81fce image is listed for CCO for example but not the other one which the pod is trying to pull.

The output of oc adm release info from the cluster lists the image digest sha256:5eec5d1cde5c71750b9b4412411b4b747fb80bed05663b8b371a0ab970663cd8 but it should be sha256:5b514efab0b38704677ef213a991960df181264c480ec7886bf251b6f2a81fce as per the the release.txt.

    4.14.18

    In customer environment

    1.
    2.
    3.
    

    Components trying to pull images with unexpected sha256 values

    Components should pull the images as per the sha256 digest values in release.txt