• Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • rhel-9.5
    • CentOS Stream 9
    • rpm-ostree
    • None
    • rpm-ostree-2024.4-3.el9
    • None
    • None
    • rhel-sst-rhcos
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • x86_64
    • None

      What were you trying to do that didn't work?

      On centos-stream9, osbuild-composer failed to build edge-container image, please find log in attachment.

      Please provide the package NVR for which bug is seen:

      [cloud-user@yih-c9-4323 rhel-edge]$ rpm -qa|grep osbuild
      python3-osbuild-111-1.el9.noarch
      osbuild-selinux-111-1.el9.noarch
      osbuild-111-1.el9.noarch
      osbuild-depsolve-dnf-111-1.el9.noarch
      osbuild-composer-core-101-1.el9.x86_64
      osbuild-ostree-111-1.el9.noarch
      osbuild-lvm2-111-1.el9.noarch
      osbuild-luks2-111-1.el9.noarch
      osbuild-composer-worker-101-1.el9.x86_64
      osbuild-composer-101-1.el9.x86_64

      How reproducible:

      100%

      Steps to reproduce

      1. Provision a centos-stream9 machine in opentstack.
      2. Login and clone test repo: git clone https://github.com/virt-s1/rhel-edge.git
      3. Run any ostree test, all failed at edge-container step: DOWNLOAD_NODE=download-node-02.eng.bos.redhat.com ./ostree-simplified-installer.sh
      4. Found error messages in log (please see the full log in attachment):

      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="    error: Postprocessing and committing: Finalizing rootfs: Hardlinking rpmdb to base location: Hardlinking /usr/share/rpm to /usr/lib/sysimage/rpm-ostree-base-db: Analyzing /usr/share/rpm/ content: File exists (os error 17)" jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="    Traceback (most recent call last):" jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="      File \"/run/osbuild/bin/org.osbuild.ostree.commit\", line 124, in <module>" jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="        r = main(args[\"inputs\"]," jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="      File \"/run/osbuild/bin/org.osbuild.ostree.commit\", line 111, in main" jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="        subprocess.run(argv," jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="      File \"/usr/lib64/python3.9/subprocess.py\", line 528, in run" jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="        raise CalledProcessError(retcode, process.args," jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="    subprocess.CalledProcessError: Command '['rpm-ostree', 'compose', 'commit', '--repo=/run/osbuild/tree/repo', '--add-metadata-string=version=9-stream', '--add-metadata-string=rpmostree.inputhash=98df5509e93f08976080b92dc9c298d701bd48c144faed033a8bc8ac01ec5099', '--write-composejson-to=/run/osbuild/tree/compose.json', '/tmp/tmp9j2j3j4w.json', '/run/osbuild/tree/tmpvcmr_q_b']' returned non-zero exit status 1." jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=info msg="    " jobId=6cf016b3-f610-4410-a7da-05550b3666db
      Mar 18 22:17:15 yih-c9-4323 osbuild-worker[23299]: time="2024-03-18T22:17:15-04:00" level=error msg="osbuild job failed: osbuild build failed" jobId=6cf016b3-f610-4410-a7da-05550b3666db

      Expected results

      test should pass

      Actual results

      test failed at edge-container step.

            [RHEL-29559] Fail to build edge-commit on centos-stream 9

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (rpm-ostree bug fix and enhancement update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHBA-2024:9096

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (rpm-ostree bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:9096

            gitlab-bot added a comment -

            gitlab-bot added a comment - Colin Walters mentioned this issue in a commit of Red Hat / Red Hat Enterprise Linux / rpms / rpm-ostree : Backport patch to fix https://issues.redhat.com/browse/RHEL-29559

            > Any idea when this fix will land in centos stream 9 repos? https://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/ still has rpm-ostree-2024.4-2.el9.x86_64.rpm

            ping

            Richard Megginson added a comment - > Any idea when this fix will land in centos stream 9 repos? https://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/ still has rpm-ostree-2024.4-2.el9.x86_64.rpm ping

            Any idea when this fix will land in centos stream 9 repos? https://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/ still has rpm-ostree-2024.4-2.el9.x86_64.rpm

            Richard Megginson added a comment - Any idea when this fix will land in centos stream 9 repos? https://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/ still has rpm-ostree-2024.4-2.el9.x86_64.rpm

            Can someone mark this Verified:Tested ?

            Colin Walters added a comment - Can someone mark this Verified:Tested ?

            I don't understand why the jira linker didn't pick this up, but https://gitlab.com/redhat/centos-stream/rpms/rpm-ostree/-/merge_requests/43 is merged and building

            Colin Walters added a comment - I don't understand why the jira linker didn't pick this up, but https://gitlab.com/redhat/centos-stream/rpms/rpm-ostree/-/merge_requests/43 is merged and building

            Micah Abbott added a comment - upstream xref: https://github.com/coreos/rpm-ostree/issues/4879#issuecomment-2012437945

            Simon de Vlieger added a comment - - edited

            The same issue is happening in Fedora rawhide (41) with `rpm-ostree-2024.4-2.fc41.aarch64`:

            ```

            error: Postprocessing and committing: Finalizing rootfs: Hardlinking rpmdb to base location: Hardlinking /usr/share/rpm to /usr/lib/sysimage/rpm-ostree-base-db: Analyzing /usr/share/rpm/ content: File exists (os error 17)

            ```

            rhn-support-yih the `rpm-ostree` version from the buildroot is used.

            Simon de Vlieger added a comment - - edited The same issue is happening in Fedora rawhide (41) with ` rpm-ostree-2024.4-2.fc41.aarch64 `: ``` error: Postprocessing and committing: Finalizing rootfs: Hardlinking rpmdb to base location: Hardlinking /usr/share/rpm to /usr/lib/sysimage/rpm-ostree-base-db: Analyzing /usr/share/rpm/ content: File exists (os error 17) ``` rhn-support-yih the `rpm-ostree` version from the buildroot is used.

            Yi He added a comment -

            I am not sure where osbuild-composer get rpm-ostree when it build images. From the host machine where osbuild-composer is running or from /etc/osbuild-composer/repositories/?

             

            I will provide both:

            1. I used this repo url  in /etc/osbuild-composer/repositories/centos-9.json, https://composes.stream.centos.org/production/latest-CentOS-Stream/compose , so the rpm-ostree version is  rpm-ostree-2024.4-2.el9.x86_64.rpm  rpm-ostree-libs-2024.4-2.el9.i686.rpm  rpm-ostree-libs-2024.4-2.el9.x86_64.rpm  
            2. This is the version on host machine:

            [root@yih-c9-4323 ~]# rpm -qa|grep rpm-ostree
            rpm-ostree-libs-2024.4-2.el9.x86_64
            rpm-ostree-2024.4-2.el9.x86_64

            Yi He added a comment - I am not sure where osbuild-composer get rpm-ostree when it build images. From the host machine where osbuild-composer is running or from /etc/osbuild-composer/repositories/?   I will provide both: I used this repo url  in /etc/osbuild-composer/repositories/centos-9.json, https://composes.stream.centos.org/production/latest-CentOS-Stream/compose , so the rpm-ostree version is   rpm-ostree-2024.4-2.el9.x86_64.rpm   rpm-ostree-libs-2024.4-2.el9.i686.rpm   rpm-ostree-libs-2024.4-2.el9.x86_64.rpm   This is the version on host machine: [root@yih-c9-4323 ~] # rpm -qa|grep rpm-ostree rpm-ostree-libs-2024.4-2.el9.x86_64 rpm-ostree-2024.4-2.el9.x86_64

            Can you provide the rpm-ostree NEVRA here as well? Thanks

            Timothée Ravier added a comment - Can you provide the rpm-ostree NEVRA here as well? Thanks

              walters@redhat.com Colin Walters
              rhn-support-yih Yi He
              CoreOS Bot CoreOS Bot
              no-user-match-found no-user-match-found
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: