Description of problem:

RHEL Enterprise Upgrade 8.9 to 9.3 fails - /boot device with UUID not found

There's a KB about:

Leapp can fail when /boot is a multipath:

https://access.redhat.com/solutions/6962799

After replacing boot-UUID /boot in fstab with /dev/md126 - Upgrade did fail again.

Version-Release number of selected component (if applicable):

leapp version 0.16.0

How reproducible:

Steps to Reproduce:

1. on RHEL Enterprise 8.9 - run "leapp upgrade"
leapp will report: reboot now
2. reboot
3. upgrade will fail - landing in emergency cli

Actual results:

no workaround seems to work

Expected results:

• RHEL upgrade procedure (pyhton scripts) should properly handle situations, where system running Multipath configurations (md), because this is common used case for many systems.

Additional info:

1rst boot upon reboot request of leapp:

[ 9.829615] localhost upgrade[1214]: mount: /boot: can't find UUID=xxxx.xxxx.xxxx.
[ 9.838088] localhost kernel: XFS (dm-8): Mounting V5 Filesystem
[ 9.994909] localhost kernel: XFS (dm-8): Ending clean mount
[ 10.004975] localhost kernel: xfs filesystem being mounted at /home supports timestamps until 2038 (0x7fffffff)
[ 32.928596] localhost upgrade[1223]: ==> Processing phase `InitRamStart`
[ 32.928596] localhost upgrade[1223]: ====> * remove_upgrade_boot_entry
[ 32.928596] localhost upgrade[1223]: Remove boot entry for Leapp provided initramfs.
[ 33.185882] localhost upgrade[1622]: Process Process-176:
[ 33.185882] localhost upgrade[1622]: Traceback (most recent call last):
[ 33.185882] localhost upgrade[1622]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
[ 33.185882] localhost upgrade[1622]: self.run()
[ 33.185882] localhost upgrade[1622]: File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
[ 33.185882] localhost upgrade[1622]: self._target(*self._args, **self._kwargs)
[ 33.185882] localhost upgrade[1622]: File "/usr/lib/python3.6/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
[ 33.185882] localhost upgrade[1622]: actor_instance.run(*args, **kwargs)
[ 33.185882] localhost upgrade[1622]: File "/usr/lib/python3.6/site-packages/leapp/actors/_init_.py", line 289, in run
[ 33.185882] localhost upgrade[1622]: self.process(*args)
[ 33.185882] localhost upgrade[1622]: File "/etc/leapp/repos.d/system_upgrade/common/actors/removeupgradebootentry/actor.py", line 20, in process
[ 33.185882] localhost upgrade[1622]: remove_boot_entry()
[ 33.185882] localhost upgrade[1622]: File "/etc/leapp/repos.d/system_upgrade/common/actors/removeupgradebootentry/libraries/removeupgradebootentry.py", line 30, in remove_boot_entry
[ 33.185882] localhost upgrade[1622]: '--remove-kernel=

'.format(kernel_filepath)
[ 33.185882] localhost upgrade[1622]: File "/usr/lib/python3.6/site-packages/leapp/libraries/stdlib/_init_.py", line 192, in run
[ 33.185882] localhost upgrade[1622]: result=result
[ 33.185882] localhost upgrade[1622]: leapp.libraries.stdlib.CalledProcessError: Command ['/usr/sbin/grubby', '--remove-kernel=/boot/vmlinuz-upgrade.x86_64'] failed with exit code 1.

2nd boot - replacing uuid of fstab /boot with md-device (/dev/md126):

[ 9.912370] localhost systemd[1]: Finished dracut pre-pivot and cleanup hook.
[ 9.960504] localhost kernel: xfs filesystem being remounted at /sysroot supports timestamps until 2038 (0x7fffffff)
[ 9.956914] localhost upgrade[1192]: Found file /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed
[ 9.957712] localhost upgrade[1192]: Error: Leapp previously failed and cannot continue, returning back to emergency shell
[ 9.957712] localhost upgrade[1192]: Please file a support case with /sysroot/var/log/leapp/leapp-upgrade.log attached
[ 9.957712] localhost upgrade[1192]: To rerun the upgrade upon exiting the dracut shell remove the /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed file
[ 9.959836] localhost upgrade[1099]: writing logs to disk and rebooting
[ 10.345738] localhost upgrade[1198]: mount: /boot: special device /dev/md126 does not exist.
[ 10.350740] localhost kernel: /dev/md126: Can't open blockdev
[ 10.352841] localhost kernel: XFS (dm-8): Mounting V5 Filesystem
[ 10.527509] localhost kernel: XFS (dm-8): Ending clean mount
[ 10.530779] localhost kernel: xfs filesystem being mounted at /home supports timestamps until 2038 (0x7fffffff)
[ 10.536318] localhost kernel: XFS (dm-8): Unmounting Filesystem
[ 10.547479] localhost upgrade[1099]: /bin/upgrade: line 19: /sysroot/var/tmp/system-upgrade.state: Read-only file system
[ 10.548229] localhost systemd[1]: upgrade.service: Main process exited, code=exited, status=1/FAILURE

output from leapp-upgrade.log:

Dec 30 12:07:34 localhost systemd[1]: Reached target Local Encrypted Volumes.
Dec 30 12:07:34 localhost systemd[1]: Reached target Path Units.
Dec 30 12:07:34 localhost systemd[1]: Starting Wait for udev To Complete Device Initialization...
Dec 30 12:07:34 localhost systemd[1]: sys-module-configfs.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit modprobe@configfs.service not found.
Dec 30 12:07:34 localhost udevadm[1536]: systemd-udev-settle.service is deprecated. Please fix multipathd-configure.service not to pull it in.
Dec 30 12:07:34 localhost systemd[1]: Finished Wait for udev To Complete Device Initialization.
Dec 30 12:07:34 localhost systemd[1]: Device-Mapper Multipath Default Configuration was skipped because of an unmet condition check (ConditionPathExists=!/etc/multipath.conf).
Dec 30 12:07:34 localhost systemd[1]: Starting Device-Mapper Multipath Device Controller...
Dec 30 12:07:34 localhost systemd[1]: Finished dracut initqueue hook.
Dec 30 12:07:34 localhost systemd[1]: Reached target Preparation for Remote File Systems.
Dec 30 12:07:34 localhost systemd[1]: Reached target Remote Encrypted Volumes.
Dec 30 12:07:34 localhost systemd[1]: Reached target Remote File Systems.
Dec 30 12:07:34 localhost systemd[1]: Starting dracut pre-mount hook...
Dec 30 12:07:34 localhost multipathd[1578]: -------start up-------
Dec 30 12:07:34 localhost multipathd[1578]: read /etc/multipath.conf
Dec 30 12:07:34 localhost multipathd[1578]: path checkers start up
Dec 30 12:07:34 localhost systemd[1]: sys-devices-virtual-block-md127.device: Failed to enqueue SYSTEMD_WANTS= job, ignoring: Unit mdmonitor.service not found.
Dec 30 12:07:34 localhost systemd[1]: Finished dracut pre-mount hook.
Dec 30 12:07:34 localhost systemd[1]: Started Device-Mapper Multipath Device Controller.
Dec 30 12:07:34 localhost systemd[1]: Reached target Initrd Root File System.
Dec 30 12:07:34 localhost systemd[1]: Reached target Preparation for Local File Systems.
Dec 30 12:07:34 localhost systemd[1]: Reached target Local File Systems.
Dec 30 12:07:34 localhost systemd[1]: Reached target System Initialization.
Dec 30 12:07:34 localhost systemd[1]: Reached target Basic System.
Dec 30 12:07:34 localhost systemd[1]: Reached target System Upgrade.
Dec 30 12:07:34 localhost systemd[1]: dracut mount hook was skipped because no trigger condition checks were met.
Dec 30 12:07:34 localhost systemd[1]: Starting dracut pre-pivot and cleanup hook...
Dec 30 12:07:34 localhost systemd[1]: Started Hardware RNG Entropy Gatherer Daemon.
Dec 30 12:07:34 localhost systemd[1]: Starting System Upgrade...
Dec 30 12:07:34 localhost upgrade[1619]: starting upgrade hook
Dec 30 12:07:34 localhost rngd[1617]: Disabling 7: PKCS11 Entropy generator (pkcs11)
Dec 30 12:07:34 localhost rngd[1617]: Disabling 5: NIST Network Entropy Beacon (nist)
Dec 30 12:07:34 localhost rngd[1617]: Disabling 9: Qrypt quantum entropy beacon (qrypt)
Dec 30 12:07:34 localhost rngd[1617]: Initializing available sources
Dec 30 12:07:34 localhost rngd[1617]: [hwrng ]: Initialization Failed
Dec 30 12:07:34 localhost rngd[1617]: [rdrand]: Initialization Failed
Dec 30 12:07:34 localhost rngd[1617]: [jitter]: JITTER timeout set to 5 sec
Dec 30 12:07:34 localhost upgrade[1629]: 7 logical volume(s) in volume group "cl_vg1" now active
Dec 30 12:07:34 localhost kernel: xfs filesystem being remounted at /sysroot supports timestamps until 2038 (0x7fffffff)
Dec 30 12:07:34 localhost upgrade[1658]: Found file /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed
Dec 30 12:07:34 localhost upgrade[1658]: Error: Leapp previously failed and cannot continue, returning back to emergency shell

Multipath.conf:
---------------

after the upgrade failed the 1rst time, i did run this command:

"mpathconf --enable"

cat /etc/multipath.conf

1. device-mapper-multipath configuration file

1. For a complete list of the default configuration values, run either:
2. # multipath -t
3. or
4. # multipathd show config

1. For a list of configuration options with descriptions, see the
2. multipath.conf man page.

defaults {
user_friendly_names yes
find_multipaths yes
enable_foreign "^$"
}

blacklist_exceptions {
property "(SCSI_IDENT_|ID_WWN)"
}

blacklist {
}