Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-63086

OCP 4.19.10 manifest digest inconsistency

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.19.z
    • oc-mirror
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • x86_64
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Customer is trying to mirror 4.19.10 in a disconnected cluster but the process is failing due to a manifest unknown. The yaml configs were already set: (bottom of the comment)

      1 - The error log

      [evuser@deployment-controller cluster-resources]$ /usr/local/bin/oc adm release extract -a /home/evuser/registry_auth.json --icsp-file=icsp-oc-mirror.yaml --insecure=true --command=openshift-install --to=/tmp mirror-registry.ems.energyvault.com:5000/openshift/release-images:4.19.10-x86_64 -v=10
      Flag --icsp-file has been deprecated, support for it will be removed in a future release. Use --idms-file instead.
      warning: --idms-file(and --icsp-file) only applies to images referenced by digest and will be ignored for tags
      I1014 16:57:04.496502 63996 round_trippers.go:473] curl -v -XGET -H "User-Agent: oc/4.19.0 (linux/amd64) kubernetes/298429b" 'https://mirror-registry.ems.energyvault.com:5000/v2/'
      I1014 16:57:04.496734 63996 round_trippers.go:502] HTTP Trace: DNS Lookup for mirror-registry.ems.energyvault.com resolved to [

      {10.0.3.50 }

      ]
      I1014 16:57:04.497000 63996 round_trippers.go:517] HTTP Trace: Dial to tcp:10.0.3.50:5000 succeed
      I1014 16:57:04.505336 63996 round_trippers.go:560] GET https://mirror-registry.ems.energyvault.com:5000/v2/ 401 Unauthorized in 8 milliseconds
      I1014 16:57:04.505360 63996 round_trippers.go:577] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 7 ms ServerProcessing 0 ms Duration 8 ms
      I1014 16:57:04.505365 63996 round_trippers.go:584] Response Headers:
      I1014 16:57:04.505371 63996 round_trippers.go:587] Date: Tue, 14 Oct 2025 16:57:04 GMT
      I1014 16:57:04.505375 63996 round_trippers.go:587] Content-Type: application/json
      I1014 16:57:04.505378 63996 round_trippers.go:587] Docker-Distribution-Api-Version: registry/2.0
      I1014 16:57:04.505383 63996 round_trippers.go:587] Www-Authenticate: Basic realm="Registry Realm"
      I1014 16:57:04.505388 63996 round_trippers.go:587] Content-Length: 87
      I1014 16:57:04.506040 63996 round_trippers.go:473] curl -v -XHEAD -H "Authorization: Basic <masked>" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.oci.image.index.v1+json" -H "Accept: application/vnd.oci.image.manifest.v1+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Accept: application/json" -H "User-Agent: oc/4.19.0 (linux/amd64) kubernetes/298429b" 'https://mirror-registry.ems.energyvault.com:5000/v2/openshift/release-images/manifests/4.19.10-x86_64'
      I1014 16:57:04.697360 63996 round_trippers.go:560] HEAD https://mirror-registry.ems.energyvault.com:5000/v2/openshift/release-images/manifests/4.19.10-x86_64 404 Not Found in 191 milliseconds
      I1014 16:57:04.697389 63996 round_trippers.go:577] HTTP Statistics: GetConnection 0 ms ServerProcessing 191 ms Duration 191 ms
      I1014 16:57:04.697400 63996 round_trippers.go:584] Response Headers:
      I1014 16:57:04.697410 63996 round_trippers.go:587] Content-Type: application/json
      I1014 16:57:04.697425 63996 round_trippers.go:587] Docker-Distribution-Api-Version: registry/2.0
      I1014 16:57:04.697435 63996 round_trippers.go:587] Content-Length: 209
      I1014 16:57:04.697443 63996 round_trippers.go:587] Date: Tue, 14 Oct 2025 16:57:04 GMT
      I1014 16:57:04.697501 63996 round_trippers.go:473] curl -v -XGET -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.oci.image.index.v1+json" -H "Accept: application/vnd.oci.image.manifest.v1+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Accept: application/json" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Basic <masked>" -H "User-Agent: oc/4.19.0 (linux/amd64) kubernetes/298429b" 'https://mirror-registry.ems.energyvault.com:5000/v2/openshift/release-images/manifests/4.19.10-x86_64'
      I1014 16:57:04.888065 63996 round_trippers.go:560] GET https://mirror-registry.ems.energyvault.com:5000/v2/openshift/release-images/manifests/4.19.10-x86_64 404 Not Found in 190 milliseconds
      I1014 16:57:04.888093 63996 round_trippers.go:577] HTTP Statistics: GetConnection 0 ms ServerProcessing 190 ms Duration 190 ms
      I1014 16:57:04.888103 63996 round_trippers.go:584] Response Headers:
      I1014 16:57:04.888114 63996 round_trippers.go:587] Content-Length: 209
      I1014 16:57:04.888123 63996 round_trippers.go:587] Date: Tue, 14 Oct 2025 16:57:04 GMT
      I1014 16:57:04.888130 63996 round_trippers.go:587] Content-Type: application/json
      I1014 16:57:04.888139 63996 round_trippers.go:587] Docker-Distribution-Api-Version: registry/2.0
      I1014 16:57:04.888272 63996 workqueue.go:143] about to send work queue error: image "mirror-registry.ems.energyvault.com:5000/openshift/release-images:4.19.10-x86_64" not found: manifest unknown: manifest unknown
      error: image "mirror-registry.ems.energyvault.com:5000/openshift/release-images:4.19.10-x86_64" not found: manifest unknown: manifest unknown
      ~~~"

      2 - Tested the openshift-install

      [evuser@deployment-controller ~]$ openshift-install agent create image --dir /data/openshift-install --log-level debug
      DEBUG OpenShift Installer 4.19.14
      DEBUG Built from commit 7e30b7d6b421087ee6b6aaa639e40392c22ce52b
      DEBUG Fetching Agent Installer ISO...
      DEBUG Loading Agent Installer ISO...
      (...)

      DEBUG Generating Agent Installer Artifacts...
      DEBUG Using mirror configuration
      DEBUG Fetching image from OCP release ([oc adm release info --image-for=agent-installer-utils --filter-by-os=linux/amd64 --insecure=true --icsp-file=/tmp/icsp-file923530913 quay.io/openshift-release-dev/ocp-release@sha256:f8e21e76897b3f9b8a76a07b5a9426ba8def9b2e56b18d8b40ad65931b8bbf78])
      DEBUG extracting /usr/bin/agent-tui to /home/evuser/.cache/agent/files_cache, [oc image extract --path=/usr/bin/agent-tui:/home/evuser/.cache/agent/files_cache --filter-by-os=linux/amd64 --insecure=true --confirm --icsp-file=/tmp/icsp-file1567689105 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d94ae4fefca3ae786a1433394a93e0af7ebb320dca48278c17b0f553899ec9f5]
      ERROR failed to write asset (Agent Installer ISO) to disk: cannot generate ISO image due to configuration errors
      FATAL failed to fetch Agent Installer ISO: failed to fetch dependency of "Agent Installer ISO": failed to generate asset "Agent Installer Artifacts": command '[oc image extract --path=/usr/bin/agent-tui:/home/evuser/.cache/agent/files_cache --filter-by-os=linux/amd64 --insecure=true --confirm --icsp-file=/tmp/icsp-file1567689105 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d94ae4fefca3ae786a1433394a93e0af7ebb320dca48278c17b0f553899ec9f5 --registry-config=/tmp/registry-config330406590]' exited with non-zero exit code 1:
      FATAL Flag --icsp-file has been deprecated, support for it will be removed in a future release. Use --idms-file instead.
      FATAL error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d94ae4fefca3ae786a1433394a93e0af7ebb320dca48278c17b0f553899ec9f5: unauthorized: access to the requested resource is not authorized
      FATAL

      3 - I tested the crictl pull in the Resource Hub cluster and it is failing. Signature Digest worked properly:

      3.1 - From the oc-mirror debug: (could not find destination entry).

      2025/10/10 14:19:34  [DEBUG]  : [ReleaseImageCollector] image manifest digest 034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9
      2025/10/10 14:19:34  [DEBUG]  : [ReleaseImageCollector] config digest
      2025/10/10 14:19:34  [DEBUG]  : extract directory exists (nop)

      3.2 - Tested the pull manually:

      crictl pull quay.io/openshift-release-dev/ocp-release@sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9
      E1014 18:34:28.200214 2053318 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = initializing source docker://quay.io/openshift-release-dev/ocp-release@sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9: reading manifest sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9 in quay.io/openshift-release-dev/ocp-release: manifest unknown" image="quay.io/openshift-release-dev/ocp-release@sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9"
      FATA[0001] pulling image: initializing source docker://quay.io/openshift-release-dev/ocp-release@sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9: reading manifest sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9 in quay.io/openshift-release-dev/ocp-release: manifest unknown

      4- The yaml configs

      apiVersion: config.openshift.io/v1
      kind: ImageTagMirrorSet
      metadata:
      annotations:
      createdAt: Wednesday, 08-Oct-25 22:33:03 UTC
      createdBy: oc-mirror v2
      oc-mirror_version: 4.19.0-202509230411.p2.g0a5c2ac.assembly.stream.el9-0a5c2ac
      name: itms-release-0
      spec:
      imageTagMirrors:

      • mirrors:
      • mirror-registry.ems.energyvault.com:5000/openshift/release-images
        source: quay.io/openshift-release-dev/ocp-release
      • mirrors:
      • mirror-registry.ems.energyvault.com:5000/openshift/release-images
        source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
        status: {}
        ~~~

      and also added your ICSP:
      ~~~
      [evuser@deployment-controller cluster-resources]$ cat icsp-oc-mirror.yaml
      imageContentSources:

      • mirrors:
      • mirror-registry.ems.energyvault.com:5000/openshift/release-images
        source: quay.io/openshift-release-dev/ocp-release
      • mirrors:
      • mirror-registry.ems.energyvault.com:5000/openshift/release-images
        source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

              rh-ee-aguidi Alex Guidi
              rhn-support-fcardoso Fabio Cardoso
              Fabio Cardoso
              None
              Nidan Gavali Nidan Gavali
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: