-
Bug
-
Resolution: Unresolved
-
Normal
-
4.20
-
None
TL,DR: cert-manager images in MicroShift 4.20 use multi-arch manifest lists, causing runtime digest mismatches or significant storage bloat in offline bootc builds.
Description of problem:
We are building bootc-based RHEL 9.6 / MicroShift 4.20 images for air-gapped (offline) environments. The cert-manager Operator manifests shipped with MicroShift 4.20 reference images using their Manifest List (multi-arch) digests. This creates a critical issue for offline builds, identical to the behavior seen with lvms-operator in OCPBUGS-54779 (and similar to OCPBUGS-51329).
Version-Release number of selected component (if applicable):
MicroShift 4.20 [root@ushift05 ~]# dnf info microshift-cert-manager Updating Subscription Management repositories.This system has release set to 9.6 and it receives updates only for this release.Last metadata expiration check: 1:48:05 ago on Thu 04 Dec 2025 03:09:09 PM CET. Installed Packages Name : microshift-cert-manager Version : 4.20.0 Release : 202510201126.p0.g1c4675a.assembly.4.20.0.el9 Architecture : x86_64 Size : 1.1 M Source : microshift-4.20.0-202510201126.p0.g1c4675a.assembly.4.20.0.el9.src.rpm Repository : @System From repo : rhocp-4.20-for-rhel-9-x86_64-rpms Summary : Cert Manager for MicroShift URL : https://github.com/openshift/microshift License : ASL 2.0 Description : [Maintainer] project: OCPBUGS, component: MicroShift : The microshift-cert-manager package provides the required manifests for the Cert Manager to be installed on MicroShif [root@ushift05 ~]# dnf info microshift-cert-manager-release-info Updating Subscription Management repositories.This system has release set to 9.6 and it receives updates only for this release.Last metadata expiration check: 1:48:36 ago on Thu 04 Dec 2025 03:09:09 PM CET. Installed Packages Name : microshift-cert-manager-release-info Version : 4.20.0 Release : 202510201126.p0.g1c4675a.assembly.4.20.0.el9 Architecture : noarch Size : 2.1 k Source : microshift-4.20.0-202510201126.p0.g1c4675a.assembly.4.20.0.el9.src.rpm Repository : @System From repo : rhocp-4.20-for-rhel-9-x86_64-rpms Summary : Release information for Cert Manager for MicroShift URL : https://github.com/openshift/microshift License : ASL 2.0 Description : [Maintainer] project: OCPBUGS, component: MicroShift : The microshift-cert-manager-release-info package provides release information files for this : release. These files contain the list of container image references used by Cert Manager : and can be used to embed those images into osbuilder blueprints or bootc containerfiles.
How reproducible:
Scenario A (Runtime Failure): If we embed images using skopeo copy docker://... dir://..., Skopeo resolves the Manifest List to the specific architecture (e.g., linux/amd64) and discards the Manifest List index. - Result: The offline storage contains the image with the AMD64 digest. - Failure: At runtime, the cert-manager deployment requests the Manifest List digest. CRI-O cannot find this digest locally, attempts to pull it from the internet, and fails with ImagePullBackOff / i/o timeout. Scenario B (Storage Bloat): If we use skopeo copy --all --preserve-digests (the current workaround), Skopeo preserves the Manifest List digest. - Result: Runtime works because the digest matches. - Failure: Skopeo downloads layers for ALL architectures (arm64, ppc64le, s390x), causing significant and unnecessary storage bloat on the target edge device.
Steps to Reproduce:
1. On a RHEL 9 / MicroShift 4.20 amd64 host. 2. Attempt to embed cert-manager images into a bootc container image. 3. Method 1: Run skopeo copy docker://<image>@<manifest-list-digest> dir://... without --all. - Result: Build succeeds, but MicroShift pod fails to start offline due to digest mismatch. 4. Method 2: Run skopeo copy --all docker://<image>@<manifest-list-digest> dir://... - Result: Runtime works, but storage usage spikes due to unused non-amd64 layers.
Actual results:
The cert-manager operator strictly enforces the multi-arch digest, forcing users to either suffer runtime failures or consume excessive disk space by mirroring all architectures.
Expected results:
MicroShift should allow the use of single-architecture local images for cert-manager without requiring the presence of the full upstream Manifest List, or provide a mechanism to pin the deployment to the local architecture's digest during installation.
Additional info:
Operator Image (Manifest List): registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:4d5e238300ce6f427a1045d51d6b37a4e5c5633985208ebb44f91e7dd53897d9 Resolved AMD64 Image: sha256:0431bde46dd25e63db6e3d2d5395d6d27f44b6e0b0bee1a4dd65137152d6c650
- is related to
-
OCPBUGS-54779 RHDE (bootc upgrade Issue) ISO Size Increase Due to Duplicate Image Storage in MicroShift Preload
-
- New
-
-
OCPBUGS-51329 MicroShift LVMS image point to a distribution manifest
-
- Closed
-