In Heterogeneous Hypershift Hosted cluster of 4.20 having power and x86 worker nodes, `oc image mirror` cmd always failed when mirroring large image to internal registry with error `time="2025-10-27T10:30:42Z" level=fatal msg="Error writing blob: Error uploading layer chunked: blob upload unknown: blob upload unknown to registry"`
But could succeed with little image, smaller than 100M, meantime, the internal registry functions well when push large image(200M+) using podman cmd.
It cannot always be reproduced in other platform and profile.
    

4.20.0-0.nightly-multi-2025-10-27-013055
    

always in Heterogeneous Hypershift Hosted cluster of 4.20 having power and x86 worker nodes
    

    1. Mirror large image in Heterogeneous Hypershift Hosted cluster
route=default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net
oc --kubeconfig=/tmp/ibm-guest image mirror $route/openshift/cli:latest $route/wxj/myimage:latest --insecure -a /tmp/auth.json --filter-by-os='.*' --keep-manifest-list
info: Planning completed in 8.15s
warning: Expected to mount sha256:41007da54c06a2015556990008a956dc4989963c9cdfd494f4055095bf233e70 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
warning: Expected to mount sha256:9d458e2e81cb0fa811f569aaf711628309c0372c7d5eed4a8ea9ec96b4aeeb42 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
warning: Expected to mount sha256:6ff3e4ed27c2a764ed399e5a7add8510208ffb867d145f1733bf64a87a15e55b from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
warning: Expected to mount sha256:d8caca94648954476f4aba702afa95c607a70840117239d5f3cf0599ef04fefe from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
warning: Expected to mount sha256:ee01206434295c146bd04f88f13f7277109373cb853fdf8f969865b0001b3c70 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
warning: Expected to mount sha256:67705065e025181e4faca8aabe1305bdd92f5bdf8a2b8009cdb69183ac2e2c47 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:6ff3e4ed27c2a764ed399e5a7add8510208ffb867d145f1733bf64a87a15e55b 20.52KiB
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:d8caca94648954476f4aba702afa95c607a70840117239d5f3cf0599ef04fefe 20.52KiB
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:9d458e2e81cb0fa811f569aaf711628309c0372c7d5eed4a8ea9ec96b4aeeb42 8.87MiB
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:41007da54c06a2015556990008a956dc4989963c9cdfd494f4055095bf233e70 72.3MiB
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:ee01206434295c146bd04f88f13f7277109373cb853fdf8f969865b0001b3c70 46.19MiB
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:67705065e025181e4faca8aabe1305bdd92f5bdf8a2b8009cdb69183ac2e2c47 47.54MiB
warning: Expected to mount sha256:71f6ba5f99438e7140ce65529ee76b60847ba7d985b3acd02577b057383162d4 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
warning: Expected to mount sha256:3f6592ee4574f8a4778120cd6fdecfa4a4166221798feeac7fd7dc1b3f343194 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:71f6ba5f99438e7140ce65529ee76b60847ba7d985b3acd02577b057383162d4 71.57MiB
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:3f6592ee4574f8a4778120cd6fdecfa4a4166221798feeac7fd7dc1b3f343194 73.19MiB
warning: Expected to mount sha256:abced7ee58448ba09b150b6a5d0083ceb67e2e834226c951cc0c678891596db9 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:abced7ee58448ba09b150b6a5d0083ceb67e2e834226c951cc0c678891596db9 73.45MiB
warning: Expected to mount sha256:39e6af120f9014592f3458a0887ae0314eafa545c2dd21a1cf1168700bf6333c from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:39e6af120f9014592f3458a0887ae0314eafa545c2dd21a1cf1168700bf6333c 70.25MiB
warning: Expected to mount sha256:d81c7fda87ac005d6fd3ebe98f0581e2f4b30cbe756770536a586ddee16d7b3e from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:d81c7fda87ac005d6fd3ebe98f0581e2f4b30cbe756770536a586ddee16d7b3e 20.51KiB
warning: Expected to mount sha256:d58592a078f0cfe5914157a53a82b3b1749ff530a4408aa5650b3639929b7366 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:d58592a078f0cfe5914157a53a82b3b1749ff530a4408aa5650b3639929b7366 46.15MiB
warning: Expected to mount sha256:6984fb942d11ae9c3798e5ffe0794d0ea47722540eb00108caf03bc3b6a67888 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:6984fb942d11ae9c3798e5ffe0794d0ea47722540eb00108caf03bc3b6a67888 20.52KiB
warning: Expected to mount sha256:d1a710e46bb5b262dc33331c329f4ec58dbc1b9d94749c0dbac11f03f7f37444 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:d1a710e46bb5b262dc33331c329f4ec58dbc1b9d94749c0dbac11f03f7f37444 81.13MiB
warning: Expected to mount sha256:1535016727f35083999f366e71bceb5b92359d089d16959257a9153c49ed2dd6 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:1535016727f35083999f366e71bceb5b92359d089d16959257a9153c49ed2dd6 67.79MiB
warning: Expected to mount sha256:c6227e48b5fd2329702f04ec0516506bd9fa6e6b04f3da5435028b1c2ba55431 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:c6227e48b5fd2329702f04ec0516506bd9fa6e6b04f3da5435028b1c2ba55431 8.751MiB
warning: Expected to mount sha256:da1f89f5a1d4adb9225e45d425834a0db0faa4597dd04f7ba2c7c3406a989a4f from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:da1f89f5a1d4adb9225e45d425834a0db0faa4597dd04f7ba2c7c3406a989a4f 51.12MiB
warning: Expected to mount sha256:25c75c34b2e2b68ba9245d9cddeb6b8a0887371ed30744064f85241a75704d87 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:25c75c34b2e2b68ba9245d9cddeb6b8a0887371ed30744064f85241a75704d87 75.59MiB
warning: Expected to mount sha256:17e86e2a16c0cf15b46286ebc1232d5d547c4d8783a227a6b3de595567d19f9b from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:17e86e2a16c0cf15b46286ebc1232d5d547c4d8783a227a6b3de595567d19f9b 8.781MiB
warning: Expected to mount sha256:c9c34288cd922d4270d7c34465d282573873739c1e5640a330fe07905cf20670 from default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/openshift/cli but mount was ignored
uploading: default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage sha256:c9c34288cd922d4270d7c34465d282573873739c1e5640a330fe07905cf20670 9.184MiB
error: unable to copy layer sha256:9d458e2e81cb0fa811f569aaf711628309c0372c7d5eed4a8ea9ec96b4aeeb42 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n"
error: unable to copy layer sha256:71f6ba5f99438e7140ce65529ee76b60847ba7d985b3acd02577b057383162d4 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n"
error: unable to copy layer sha256:3f6592ee4574f8a4778120cd6fdecfa4a4166221798feeac7fd7dc1b3f343194 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n"
error: unable to copy layer sha256:ee01206434295c146bd04f88f13f7277109373cb853fdf8f969865b0001b3c70 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n"
error: unable to copy layer sha256:abced7ee58448ba09b150b6a5d0083ceb67e2e834226c951cc0c678891596db9 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n"
error: unable to copy layer sha256:d58592a078f0cfe5914157a53a82b3b1749ff530a4408aa5650b3639929b7366 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n"
error: unable to copy layer sha256:39e6af120f9014592f3458a0887ae0314eafa545c2dd21a1cf1168700bf6333c to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: Patch "https://default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/v2/wxj/myimage/blobs/uploads/ebebce4e-14f5-4b37-b70a-755a11b31bbf?_state=W3io8mcUCnO4Uz5CKBlFPb-WwjKokP8U5lZMsyBdrrR7Ik5hbWUiOiJ3eGovbXlpbWFnZSIsIlVVSUQiOiJlYmViY2U0ZS0xNGY1LTRiMzctYjcwYS03NTVhMTFiMzFiYmYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTAtMzBUMTM6MjQ6NDIuNzQ4NDE5ODc1WiJ9": unexpected EOF
error: unable to copy layer sha256:da1f89f5a1d4adb9225e45d425834a0db0faa4597dd04f7ba2c7c3406a989a4f to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n"
error: unable to copy layer sha256:41007da54c06a2015556990008a956dc4989963c9cdfd494f4055095bf233e70 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: Patch "https://default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/v2/wxj/myimage/blobs/uploads/187623b1-f5c7-43e1-87f6-c6395f8b3fd5?_state=rTmSZzfTkecyRsZBrKbwmtDLq5ua9USQQgfw7W1usoZ7Ik5hbWUiOiJ3eGovbXlpbWFnZSIsIlVVSUQiOiIxODc2MjNiMS1mNWM3LTQzZTEtODdmNi1jNjM5NWY4YjNmZDUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTAtMzBUMTM6MjQ6MDUuNjQ2ODM0MzIyWiJ9": unexpected EOF
error: unable to copy layer sha256:d1a710e46bb5b262dc33331c329f4ec58dbc1b9d94749c0dbac11f03f7f37444 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: Patch "https://default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/v2/wxj/myimage/blobs/uploads/94a4290d-084b-4ef4-a8b8-d548da3cb17c?_state=Aul8BTA8CqvBPDH7MI2W0_C-tXR1AYrKUBXTIDt5GeZ7Ik5hbWUiOiJ3eGovbXlpbWFnZSIsIlVVSUQiOiI5NGE0MjkwZC0wODRiLTRlZjQtYThiOC1kNTQ4ZGEzY2IxN2MiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTAtMzBUMTM6MjU6MDguNTAzNTc3ODRaIn0%3D": unexpected EOF
error: unable to copy layer sha256:c9c34288cd922d4270d7c34465d282573873739c1e5640a330fe07905cf20670 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: Patch "https://default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/v2/wxj/myimage/blobs/uploads/54337b0a-8e5a-40f7-a9a3-a79553eec047?_state=sAAGT-XY_T18Am7iMFM9L2Vp3oGJeB_FMzqkUBUdyR57Ik5hbWUiOiJ3eGovbXlpbWFnZSIsIlVVSUQiOiI1NDMzN2IwYS04ZTVhLTQwZjctYTlhMy1hNzk1NTNlZWMwNDciLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTAtMzBUMTM6MzQ6MzMuNjAxMjQ4MDY3WiJ9": unexpected EOF
error: unable to copy layer sha256:25c75c34b2e2b68ba9245d9cddeb6b8a0887371ed30744064f85241a75704d87 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: Patch "https://default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/v2/wxj/myimage/blobs/uploads/9a360481-69ac-40e1-9031-0a770dca8a84?_state=tt_y5dAeeKs--f5heO7-Qh1uKFRCdiVE3MY-ynHmQhF7Ik5hbWUiOiJ3eGovbXlpbWFnZSIsIlVVSUQiOiI5YTM2MDQ4MS02OWFjLTQwZTEtOTAzMS0wYTc3MGRjYThhODQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTAtMzBUMTM6Mjg6MDcuNjA3MzkxNjA0WiJ9": unexpected EOF
error: unable to copy layer sha256:67705065e025181e4faca8aabe1305bdd92f5bdf8a2b8009cdb69183ac2e2c47 to default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/wxj/myimage: Patch "https://default-route-openshift-image-registry.apps.hypershift-600.qe-ppc64le.cis.ibm.net/v2/wxj/myimage/blobs/uploads/a9632076-6e25-43cd-b811-65b9f5a7ccd6?_state=V7PWzW5mUug3QbyhyHnE-rznJeLASUJuXLQPEcv310l7Ik5hbWUiOiJ3eGovbXlpbWFnZSIsIlVVSUQiOiJhOTYzMjA3Ni02ZTI1LTQzY2QtYjgxMS02NWI5ZjVhN2NjZDYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjUtMTAtMzBUMTM6MjQ6MDYuMzAzNzAwNTQ3WiJ9": unexpected EOF
info: Mirroring completed in 30m36.56s (48.76kB/s)
error: one or more errors occurred while uploading images
   2.
    3.
    

Could mirror large image to internal registry  using 'oc image mi rror' cmd