Normal Description of problem:
Using oc-mirror to publish images from a local mirror archive to an internal mirror registry fails when trying to upload a specific image (registry.redhat.io/ubi8/ubi-micro:latest which is 8.10-13 at time of writing - probably not related to actual image) with the following warnings and errors for multiple layers:
warning: Layer size mismatch for sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7: had 10919446, wrote 21775222
[...]
error: failed to commit blob sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7 from manifest sha256:c051ea0c253cd65e9c676db874a5290720dd4d27e13496655937e38ff16f06b3 to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: blob upload invalid: blob upload invalid
[...]
error: unable to push manifest to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro:latest manifest invalid: manifest invalid
Extracting the mirror archive indeed shows that the stored blobs were larger than expected by the manifest (21775222 B instead of 10919446 B for the example blob).
Version-Release number of selected component (if applicable):
4.16.0-202408090908.p0.g819f011.assembly.stream.el9-819f011
How reproducible:
Sporadic
Steps to Reproduce:
Unknown - occurred during a normal mirroring operation.
Steps undertaken:
1. Create imageset with .additionalImages pointing to registry.redhat.io/ubi8/ubi-micro:latest
2. Run oc-mirror with the following command to create a local mirror archive:
time oc-mirror -v 3 --config ./imageset-config.yaml file://mirror
3. Run oc-mirror with the following command to publish the mirror archive to the mirror registry
oc-mirror --from=mirror/mirror_seq6_000000.tar docker://mirror-registry.local:8443/$QUAY_ORGANIZATION
Actual results:
Publishing fails with `error: error occurred during image processing: error running generic image mirror: one or more errors occurred` and the above warnings/errors
Expected results:
1. Mirroring to local mirror archive creates correct size blobs 2. Publishing from mirror archive to mirror registry completes successfully
Additional info:
Downloading the affected image, retagging it and pushing it manually (to a custom repository) via podman seems to work. Creating a completely new mirror-archive (non sequential) with the same imageset config created a valid mirror-archive with the correct blob sizes. As a workaround, taking the newly mirrored blobs and replacing the ones in the original, sequential mirror-archive resolved the issue and allowed the mirroring to proceed
The imageset:
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
local:
path: ./oc-mirror-metadata
mirror:
platform:
graph: true
architectures:
- amd64
channels:
- name: eus-4.14
minVersion: '4.14.18'
maxVersion: '4.14.18'
shortestPath: true
type: ocp
- name: stable-4.15
minVersion: '4.15.34'
maxVersion: '4.15.34'
shortestPath: true
type: ocp
- name: eus-4.16
minVersion: '4.14.18'
maxVersion: '4.16.14'
shortestPath: true
type: ocp
additionalImages:
- name: registry.redhat.io/ubi8/ubi-micro:latest
- name: registry.redhat.io/ubi9/ubi-micro:latest
Unfortunately, the log from the mirroring of the original mirror archive is unavailable.
What follows is the full log from the failed publishing of the original mirror archive.
Checking push permissions for mirror-registry.local:8443 Publishing image set from archive "mirror/mirror_seq6_000000.tar" to registry "mirror-registry.local:8443" mirror-registry.local:8443/ ocp-releases/ubi8/ubi-micro blobs: file://ubi8/ubi-micro sha256:ec4fe632a4ca53f621b2758607eccb893d0c6981c89e3af62099dcaad4a538ea 3.923KiB file://ubi8/ubi-micro sha256:02d0d3ad0fe273281aaeed3f8bac66115ee107e4bbf0707834d2c82d06c3ea03 3.924KiB file://ubi8/ubi-micro sha256:eb537117b6f6c0c262c35f4b15159f4a8ad41efa0216fcde7fa8747fe51a9456 3.925KiB file://ubi8/ubi-micro sha256:5a640775c92b65f2c0c0d2e1f3027ceaa19c4b8f991b20083a55c2d42878faca 3.928KiB file://ubi8/ubi-micro sha256:fe7d5a6e5909199060e0d61fce31e754c5066937d8cbc586fd1b7a21f2af9217 10.34MiB file://ubi8/ubi-micro sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7 10.41MiB file://ubi8/ubi-micro sha256:04e9b930b8029951d041f3bac4b41741335c00d48f9bec1ee577d52ce66ffe75 11.3MiB file://ubi8/ubi-micro sha256:5ad47570a52a79cefd5aa76cc55cca7da3a81413c6d643e2d95fc08a063a09b3 12.92MiB manifests: sha256:189ccac002d01ef3a0504609cb317032ae501618388292ab5221662230fc70fc sha256:3cdbf3c83c50325703b01c654612f4f65ac4b626bb595eeac8a4bfce1b435a71 sha256:8996eafc36e4d02b352fe738f5998c125f8edc571f76f41b20b177f810759ace sha256:c051ea0c253cd65e9c676db874a5290720dd4d27e13496655937e38ff16f06b3 sha256:cf8d8796a2c07577b4adc8b598aa648b200cc6b4b5f78d2e13539711ea64f275 sha256:cf8d8796a2c07577b4adc8b598aa648b200cc6b4b5f78d2e13539711ea64f275 -> latest stats: shared=0 unique=8 size=44.98MiB ratio=1.00phase 0: mirror-registry.local:8443 ocp-releases/ubi8/ubi-micro blobs=8 mounts=0 manifests=6 shared=0info: Planning completed in 310ms uploading: mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro sha256:fe7d5a6e5909199060e0d61fce31e754c5066937d8cbc586fd1b7a21f2af9217 10.34MiB warning: Layer size mismatch for sha256:fe7d5a6e5909199060e0d61fce31e754c5066937d8cbc586fd1b7a21f2af9217: had 10837392, wrote 13583325 uploading: mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7 10.41MiB warning: Layer size mismatch for sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7: had 10919446, wrote 21775222 uploading: mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro sha256:04e9b930b8029951d041f3bac4b41741335c00d48f9bec1ee577d52ce66ffe75 11.3MiB warning: Layer size mismatch for sha256:04e9b930b8029951d041f3bac4b41741335c00d48f9bec1ee577d52ce66ffe75: had 11851545, wrote 16557426 sha256:3cdbf3c83c50325703b01c654612f4f65ac4b626bb595eeac8a4bfce1b435a71 mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro error: failed to commit blob sha256:fe7d5a6e5909199060e0d61fce31e754c5066937d8cbc586fd1b7a21f2af9217 from manifest sha256:c051ea0c253cd65e9c676db874a5290720dd4d27e13496655937e38ff16f06b3 to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: blob upload invalid: blob upload invalid error: failed to commit blob sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7 from manifest sha256:c051ea0c253cd65e9c676db874a5290720dd4d27e13496655937e38ff16f06b3 to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: blob upload invalid: blob upload invalid error: failed to commit blob sha256:04e9b930b8029951d041f3bac4b41741335c00d48f9bec1ee577d52ce66ffe75 from manifest sha256:c051ea0c253cd65e9c676db874a5290720dd4d27e13496655937e38ff16f06b3 to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: blob upload invalid: blob upload invalid error: unable to push manifest to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: manifest invalid: manifest invalid error: unable to push manifest to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: manifest invalid: manifest invalid error: unable to push manifest to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: manifest invalid: manifest invalid error: unable to push manifest to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro:latest: manifest invalid: manifest invalid info: Mirroring completed in 650ms (0B/s) mirror-registry.local:8443/ ocp-releases/ubi8/ubi-micro blobs: file://ubi8/ubi-micro sha256:ec4fe632a4ca53f621b2758607eccb893d0c6981c89e3af62099dcaad4a538ea 3.923KiB file://ubi8/ubi-micro sha256:02d0d3ad0fe273281aaeed3f8bac66115ee107e4bbf0707834d2c82d06c3ea03 3.924KiB file://ubi8/ubi-micro sha256:eb537117b6f6c0c262c35f4b15159f4a8ad41efa0216fcde7fa8747fe51a9456 3.925KiB file://ubi8/ubi-micro sha256:5a640775c92b65f2c0c0d2e1f3027ceaa19c4b8f991b20083a55c2d42878faca 3.928KiB file://ubi8/ubi-micro sha256:fe7d5a6e5909199060e0d61fce31e754c5066937d8cbc586fd1b7a21f2af9217 10.34MiB file://ubi8/ubi-micro sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7 10.41MiB file://ubi8/ubi-micro sha256:04e9b930b8029951d041f3bac4b41741335c00d48f9bec1ee577d52ce66ffe75 11.3MiB file://ubi8/ubi-micro sha256:5ad47570a52a79cefd5aa76cc55cca7da3a81413c6d643e2d95fc08a063a09b3 12.92MiB manifests: sha256:189ccac002d01ef3a0504609cb317032ae501618388292ab5221662230fc70fc sha256:3cdbf3c83c50325703b01c654612f4f65ac4b626bb595eeac8a4bfce1b435a71 sha256:8996eafc36e4d02b352fe738f5998c125f8edc571f76f41b20b177f810759ace sha256:c051ea0c253cd65e9c676db874a5290720dd4d27e13496655937e38ff16f06b3 sha256:cf8d8796a2c07577b4adc8b598aa648b200cc6b4b5f78d2e13539711ea64f275 sha256:cf8d8796a2c07577b4adc8b598aa648b200cc6b4b5f78d2e13539711ea64f275 -> latest stats: shared=0 unique=8 size=44.98MiB ratio=1.00phase 0: mirror-registry.local:8443 ocp-releases/ubi8/ubi-micro blobs=8 mounts=0 manifests=6 shared=0info: Planning completed in 320ms uploading: mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7 10.41MiB uploading: mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro sha256:fe7d5a6e5909199060e0d61fce31e754c5066937d8cbc586fd1b7a21f2af9217 10.34MiB warning: Layer size mismatch for sha256:fe7d5a6e5909199060e0d61fce31e754c5066937d8cbc586fd1b7a21f2af9217: had 10837392, wrote 13583325 warning: Layer size mismatch for sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7: had 10919446, wrote 21775222 uploading: mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro sha256:04e9b930b8029951d041f3bac4b41741335c00d48f9bec1ee577d52ce66ffe75 11.3MiB warning: Layer size mismatch for sha256:04e9b930b8029951d041f3bac4b41741335c00d48f9bec1ee577d52ce66ffe75: had 11851545, wrote 16557426 sha256:3cdbf3c83c50325703b01c654612f4f65ac4b626bb595eeac8a4bfce1b435a71 mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro error: failed to commit blob sha256:fe7d5a6e5909199060e0d61fce31e754c5066937d8cbc586fd1b7a21f2af9217 from manifest sha256:8996eafc36e4d02b352fe738f5998c125f8edc571f76f41b20b177f810759ace to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: blob upload invalid: blob upload invalid error: failed to commit blob sha256:f2d7b3a5ca3f58305289a6bae2b2d13c8a512bc97bc6725c33db31495a0612c7 from manifest sha256:8996eafc36e4d02b352fe738f5998c125f8edc571f76f41b20b177f810759ace to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: blob upload invalid: blob upload invalid error: failed to commit blob sha256:04e9b930b8029951d041f3bac4b41741335c00d48f9bec1ee577d52ce66ffe75 from manifest sha256:8996eafc36e4d02b352fe738f5998c125f8edc571f76f41b20b177f810759ace to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: blob upload invalid: blob upload invalid error: unable to push manifest to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: manifest invalid: manifest invalid error: unable to push manifest to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: manifest invalid: manifest invalid error: unable to push manifest to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro: manifest invalid: manifest invalid error: unable to push manifest to mirror-registry.local:8443/ocp-releases/ubi8/ubi-micro:latest: manifest invalid: manifest invalid info: Mirroring completed in 810ms (0B/s) mirror-registry.local:8443/ ocp-releases/ubi9/ubi-micro manifests: sha256:11b5e26e24ce14b02372860115162e81ae011b748619b371f261e1e97d4cf2bf sha256:4f697bc86567c377a6ce66ec8e53236de10192f53d9dbeb87b3228786648c61a sha256:7f376b75faf8ea546f28f8529c37d24adcde33dca4103f4897ae19a43d58192b sha256:d5742967ec51c18cfc583c54f8f0fa61644051b13e27e700e91dea6565fb31fe sha256:d900d78012611118c935858a0f765e70cc8de0229baa033ac1e5dead2b1ff827 sha256:7f376b75faf8ea546f28f8529c37d24adcde33dca4103f4897ae19a43d58192b -> latest stats: shared=0 unique=0 size=0Bphase 0: mirror-registry.local:8443 ocp-releases/ubi9/ubi-micro blobs=0 mounts=0 manifests=6 shared=0info: Planning completed in 320ms sha256:4f697bc86567c377a6ce66ec8e53236de10192f53d9dbeb87b3228786648c61a mirror-registry.local:8443/ocp-releases/ubi9/ubi-micro sha256:11b5e26e24ce14b02372860115162e81ae011b748619b371f261e1e97d4cf2bf mirror-registry.local:8443/ocp-releases/ubi9/ubi-micro sha256:d5742967ec51c18cfc583c54f8f0fa61644051b13e27e700e91dea6565fb31fe mirror-registry.local:8443/ocp-releases/ubi9/ubi-micro sha256:d900d78012611118c935858a0f765e70cc8de0229baa033ac1e5dead2b1ff827 mirror-registry.local:8443/ocp-releases/ubi9/ubi-micro sha256:7f376b75faf8ea546f28f8529c37d24adcde33dca4103f4897ae19a43d58192b mirror-registry.local:8443/ocp-releases/ubi9/ubi-micro:latest info: Mirroring completed in 50ms (0B/s) error: error occurred during image processing: error running generic image mirror: one or more errors occurred
