Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-23449

[Leapp] Invisible /usr in system_overlay mounts

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • rhel-8.10.z
    • leapp-repository
    • None
    • Important
    • rhel-sst-upgrades
    • 3
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • All
    • None

      What were you trying to do that didn't work?

      Leapp fails during the dnf_transaction_check, DNF being unable to load RPMs located in /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms on a separate /usr partition. RPMs in question were there on the host filesystem, but not available on system_overlay after a set of overlays + bind mounts.

      The problem is very uncommon, as our customer configured the mount propagation flag to private/slave instead of shared. AFAIK, systemd sets all FS as shared after the switch root, and does not offer any mechanism to change it. We hence suppose it's due to a 3rd party, or an hardening procedure.

      It likely also affects RHEL 8.

      Please provide the package NVR for which bug is seen:

      leapp-upgrade-el7toel8-0.19.0-1.el7_9.noarch

      How reproducible:

      Always

      Steps to reproduce

      Configure the propagation flag as slave

      # for mnt in $(awk '{print $2}' /proc/mounts); do mount -o remount --make-slave $mnt; done

      Check the behaviour with `findmnt -o TARGET,PROPAGATION`.

      Run a leapp upgrade.

      Alternatively, it can be reproduced outside of Leapp with the below set of commands

      mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_
dd if=/dev/zero of=/var/lib/leapp/xfs_root.img bs=1k count=0 seek=10M
mkfs.xfs -f /var/lib/leapp/xfs_root.img
mount -o loop /var/lib/leapp/xfs_root.img /var/lib/leapp/scratch/mounts/root_
mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_/{system_overlay,upper,work}
mount -t overlay overlay2 -o lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work /var/lib/leapp/scratch/mounts/root_/system_overlay
mkdir -m 777 /var/lib/leapp/el8userspace/installroot
mount -o bind /var/lib/leapp/scratch/mounts/root_/system_overlay /var/lib/leapp/el8userspace/installroot
mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_usr
dd if=/dev/zero of=/var/lib/leapp/xfs_usr.img bs=1k count=0 seek=10M
mkfs.xfs -f /var/lib/leapp/xfs_usr.img
mount -o loop /var/lib/leapp/xfs_usr.img /var/lib/leapp/scratch/mounts/root_usr
mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_usr/{root_usr,upper,work}
mount -t overlay overlay2 -o lowerdir=/usr,upperdir=/var/lib/leapp/scratch/mounts/root_usr/upper,workdir=/var/lib/leapp/scratch/mounts/root_usr/work /var/lib/leapp/scratch/mounts/root_usr/root_usr
mkdir -m 777 -p /var/lib/leapp/scratch/mounts/root_/system_overlay/usr
mount -o bind /var/lib/leapp/scratch/mounts/root_usr/root_usr /var/lib/leapp/scratch/mounts/root_/system_overlay/usr
ls /var/lib/leapp/el8userspace/installroot/usr
<EMPTY>

      Expected results

      Maybe an inhibitor pointing to the problem that was very complex to identify.

      Actual results

      # findmnt -o TARGET,PROPAGATION
TARGET                                PROPAGATION
/                                     private,slave
├─/usr                                private,slave

2024-01-18 15:15:14.841809 [ERROR] Actor: dnf_transaction_check
Message: DNF execution failed with non zero exit code.
Summary:
    Stderr: Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
            Failed to create directory /var/lib/leapp/el8userspace//sys/fs/selinux: Read-only file system
            No matches found for the following disable plugin patterns: subscription-manager
            Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202301271308Z.f7c82a24.master.el8.noarch.rpm.
            Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-repository-deps-el8-5.0.8-100.202301271308Z.f7c82a24.master.el8.noarch.rpm.
            Can not load RPM file: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/kernel-workaround-0.1-1.el8.noarch.rpm.
            Could not open: /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-deps-el8-5.0.8-100.202301271308Z.f7c82a24.master.el8.noarch.rpm /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/leapp-repository-deps-el8-5.0.8-100.202301271308Z.f7c82a24.master.el8.noarch.rpm /installroot/usr/share/leapp-repository/repositories/system_upgrade/el7toel8/files/bundled-rpms/kernel-workaround-0.1-1.el8.noarch.rpm

      Additional information

      Explanation on shared/slave/private propagation flag:

      https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt

      Workaround:

      # for mnt in $(awk '{print $2}' /proc/mounts); do mount -o remount --make-shared $mnt; done

              leapp-notifications leapp-notifications
              rhn-support-cbesson Christophe Besson
              leapp-notifications leapp-notifications
              RHEL Upgrades QE Team RHEL Upgrades QE Team
              Miriam Portman Miriam Portman
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: