Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-23449

[Leapp] Invisible /usr in system_overlay mounts

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • rhel-7.9.z, rhel-8.10.z
    • leapp-repository
    • Major
    • sst_upgrades
    • 3
    • False
    • Hide

      None

      Show
      None
    • None
    • All
    • None

      What were you trying to do that didn't work?

      Leapp fails during the dnf_transaction_check, DNF being unable to load RPMs located in /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms on a separate /usr partition. RPMs in question were there on the host filesystem, but not available on system_overlay after a set of overlays + bind mounts.

      The problem is very uncommon, as our customer configured the mount propagation flag to private/slave instead of shared. AFAIK, systemd sets all FS as shared after the switch root, and does not offer any mechanism to change it. We hence suppose it's due to a 3rd party, or an hardening procedure.

      It likely also affects RHEL 8.

      Please provide the package NVR for which bug is seen:

      leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch

      How reproducible:

      Always

      Steps to reproduce

      Configure the propagation flag as slave

      # for mnt in $(awk '{print $2}' /proc/mounts); do mount -o remount --make-slave $mnt; done 

      Check the behaviour with `findmnt -o TARGET,PROPAGATION`.

      Run a leapp upgrade.

      Alternatively, it can be reproduced outside of Leapp with the below set of commands

      mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_
      dd if=/dev/zero of=/var/lib/leapp/xfs_root.img bs=1k count=0 seek=10M
      mkfs.xfs -f /var/lib/leapp/xfs_root.img
      mount -o loop /var/lib/leapp/xfs_root.img /var/lib/leapp/scratch/mounts/root_
      mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_/{system_overlay,upper,work}
      mount -t overlay overlay2 -o lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work /var/lib/leapp/scratch/mounts/root_/system_overlay
      mkdir -m 777 /var/lib/leapp/el8userspace/installroot
      mount -o bind /var/lib/leapp/scratch/mounts/root_/system_overlay /var/lib/leapp/el8userspace/installroot
      mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_usr
      dd if=/dev/zero of=/var/lib/leapp/xfs_usr.img bs=1k count=0 seek=10M
      mkfs.xfs -f /var/lib/leapp/xfs_usr.img
      mount -o loop /var/lib/leapp/xfs_usr.img /var/lib/leapp/scratch/mounts/root_usr
      mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_usr/{root_usr,upper,work}
      mount -t overlay overlay2 -o lowerdir=/usr,upperdir=/var/lib/leapp/scratch/mounts/root_usr/upper,workdir=/var/lib/leapp/scratch/mounts/root_usr/work /var/lib/leapp/scratch/mounts/root_usr/root_usr
      mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_/system_overlay/usr
      mount -o bind /var/lib/leapp/scratch/mounts/root_usr/root_usr /var/lib/leapp/scratch/mounts/root_/system_overlay/usr
      ls /var/lib/leapp/el8userspace/installroot/usr
      <EMPTY> 

      Expected results

      Maybe an inhibitor pointing to the problem that was very complex to identify.

      Actual results

      # findmnt -o TARGET,PROPAGATION
      TARGET                                PROPAGATION
      /                                     private,slave
      ├─/usr                                private,slave
      
      2024-01-18 15:15:14.841809 [ERROR] Actor: dnf_transaction_check
      Message: DNF execution failed with non zero exit code.
      Summary:
          Stderr: Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
                  Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
                  No matches found for the following disable plugin patterns: subscription-manager
                  Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202301271308Z.f7c82a24.master.el8.noarch.rpm.
                  Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-repository-deps-el8-5.0.8-100.202301271308Z.f7c82a24.master.el8.noarch.rpm.
                  Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/kernel-workaround-0.1-1.el8.noarch.rpm.
                  Could not open: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202301271308Z.f7c82a24.master.el8.noarch.rpm /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-repository-deps-el8-5.0.8-100.202301271308Z.f7c82a24.master.el8.noarch.rpm /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/kernel-workaround-0.1-1.el8.noarch.rpm 

      Additional information

      Explanation on shared/slave/private propagation flag:

      https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt

      Workaround:

      # for mnt in $(awk '{print $2}' /proc/mounts); do mount -o remount --make-shared $mnt; done 

            leapp-notifications leapp-notifications
            rhn-support-cbesson Christophe Besson
            leapp-notifications leapp-notifications
            RHEL Upgrades QE Team RHEL Upgrades QE Team
            Miriam Portman Miriam Portman
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: