What were you trying to do that didn't work?

Build rhel-bootc base image with standard manifest twice with same inputs, but the generated images diff in layers

What is the impact of this issue to you?

It block reproducible container image feature for rhel image mode

Please provide the package NVR for which the bug is seen:

https://gitlab.com/redhat/rhel/bifrost/rhel-bootc/-/tree/301605de4f273bf340166ade36747bd9fddaa3b1

rpm-ostree-2025.10-1.el10

How reproducible is this bug?:

100%

Steps to reproduce

Chunked base image

1. On a rhel-10.1 host, git clone the rhel-bootc:10.1 konflux definition from https://gitlab.com/redhat/rhel/bifrost/rhel-bootc/-/tree/rhel-10.1
2. Add `ARG SOURCE_DATE_EPOCH` to the Containerfile
3. Build base image twice

   # export SOURCE_DATE_EPOCH=0
   # podman build --no-cache --security-opt=label=disable --cap-add=all --device /dev/fuse --volume $(pwd):/buildcontext -f Containerfile -t rhel-bootc 

1. Preserve the out.ociarchive each time for the inspection, compare the oci.image.manifest between those two ociarchives

    "layers": [
        {
   -      "digest": "sha256:9a3383a15d0a0d49790cf274e697280fa19b8ee7ff8b2a05f82f83476b50a63b",
   +      "digest": "sha256:66875e118e12daaf1cc07e883edaf9ee7ec4a649dbe1ef6fb8d0a74c783fb89d",
          "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
   -      "size": 1931961
   +      "size": 1931968
        },
        {
          "annotations": {
   @@ -51,9 +51,9 @@
          "annotations": {
            "ostree.components": "python3-libs"
          },
   -      "digest": "sha256:702ccdbea2a9d02bee07adca4203c2aecd5a6f0643e45fcd507c583ed3c7cb9b",
   +      "digest": "sha256:801f1c51e1d1e30bf64c4f4c6af1fef1bbd0782128bac4efa5f0f4b25611001b",
          "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
   -      "size": 17355158
   +      "size": 17355032
        },
        {
          "annotations": {
   @@ -339,9 +339,9 @@
          "annotations": {
            "ostree.components": "NetworkManager,bootupd,nvme-cli,openssl-libs,python3-policycoreutils,realtek-firmware,sssd-common,tiwilink-firmware"
          },
   -      "digest": "sha256:e00614ade0ad2d597559ec62541f0a2a6419301ea17e037a23e9a1131763493e",
   +      "digest": "sha256:4668ede84f0ea802c61a5d76d4c1aab890dc575c263c2bc80c572906d53773e4",
          "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
   -      "size": 26759108
   +      "size": 26759260
        }, 

1. Unpack the 3 diff layers to find out what exactly different

   # python3-lib layer diff in 1 file
   usr/lib64/python3.12/urllib/__pycache__/__init__.cpython-312.pyc
   # this file have a different checksum from its origin rpm package
   
   # NetworkManager layer diff in 1 file
   usr/lib/python3.12/site-packages/sepolgen/__pycache__/__init__.cpython-312.pyc
   # this file have a different checksum from its origin rpm package
   
   # The first layer diff in many ostree dirtree files and a commit file, it's all related to the two pyc file diffs I mentioned in previous two layers

Non-chunked single layer base image

1. Continue from the chunked base image workspace, comment the chunker steps

   diff --git a/Containerfile b/Containerfile
   index 19925d6..1eb716c 100644
   --- a/Containerfile
   +++ b/Containerfile
   @@ -5,6 +5,7 @@ # We bootstrap from UBI
    FROM registry.stage.redhat.io/ubi10/ubi:latest as builder
   +ARG SOURCE_DATE_EPOCH
    # Ensure we only use repos we have here, plus the cachi2 ones if set in Konflux
    RUN rm -vf /etc/yum.repos.d/ubi.repo /etc/yum.repos.d/rhel-*.repo
    COPY *.repo /etc/yum.repos.d
   @@ -16,10 +17,14 @@ WORKDIR /src
    RUN --mount=type=cache,target=/workdir \
          ./build.sh-FROM oci-archive:./out.ociarchive
   +# FROM oci-archive:./out.ociarchive
    # Need to reference builder here to force ordering. But since we have to run
    # something anyway, we might as well cleanup after ourselves.
   -RUN --mount=type=bind,from=builder,target=/var/tmp rm -v /buildcontext/out.ociarchive
   +# RUN --mount=type=bind,from=builder,target=/var/tmp rm -v /buildcontext/out.ociarchive
   +
   +FROM scratch
   +# Copy the root file system built in the previous step into this image.
   +COPY --from=builder /target-rootfs/ / # This is updated by renovate
    LABEL redhat.compose-id="RHEL-10.1-20251021.0"
   diff --git a/build.sh b/build.sh
   index 77c5112..b9fb0cf 100755
   --- a/build.sh
   +++ b/build.sh
   @@ -17,4 +17,4 @@ cat /etc/yum.repos.d/*.repo
    # Note this invocation uses /repos
    /usr/libexec/bootc-base-imagectl build-rootfs --reinject --manifest=${MANIFEST} /target-rootfs
    # Now for this we still rely on using a buildah version that predates https://github.com/containers/buildah/issues/5952
   -rpm-ostree experimental compose build-chunked-oci --bootc --format-version=1 --rootfs /target-rootfs --output oci-archive:/buildcontext/out.ociarchive
   +# rpm-ostree experimental compose build-chunked-oci --bootc --format-version=2 --rootfs /target-rootfs --output oci-archive:/buildcontext/out.ociarchive
    

1. Build base image twice

   # export SOURCE_DATE_EPOCH=0
   # podman build --no-cache --security-opt=label=disable --cap-add=all --device /dev/fuse --volume $(pwd):/buildcontext -f Containerfile -t rhel-bootc 

1. Save the container images in oci-dir and unpack the giant single layer, and compare the layer diff

   # diff -rq --no-dereference r1 r2
   Files r1/usr/lib/python3.12/site-packages/sepolgen/__pycache__/__init__.cpython-312.pyc and r2/usr/lib/python3.12/site-packages/sepolgen/__pycache__/__init__.cpython-312.pyc differ
   Files r1/usr/lib64/python3.12/urllib/__pycache__/__init__.cpython-312.pyc and r2/usr/lib64/python3.12/urllib/__pycache__/__init__.cpython-312.pyc differ 

Expected results

rhel-bootc base image is reproducible when gave same inputs for building

Actual results

Two pyc file break the reproducibility of rhel-bootc base image, it happens with only `bootc-base-imagectl build-rootfs` which didn't involve build-chunked-oci.

I'm not sure if this is a rpm-ostree issue since `bootc-base-imagectl build-rootfs` wrapped many things, so I reported against rhel-bootc-container.