-
Bug
-
Resolution: Won't Do
-
Minor
-
None
-
rhel-7.9.z
-
None
-
Low
-
sst_upgrades
-
3
-
False
-
-
None
-
None
-
None
-
None
-
If docs needed, set a value
-
-
All
-
None
Description of problem:
Leapp failed at the preupgrade step due to a directory that cannot be created on top of <overlayfs>/tmp.
The issue was not with this specific partition, if this one is not used it fails on the next one.
- They use the latest el7 kernel and the latest leapp packages
- They have no 3rd party tainting the kernel.
- All their XFS have an ftype (hence it supposedly supports overlays).
- The strace shows the first mkdir() on the mounted overlay fails with an EINVAL errno (excerpt of relevant calls):
24271 11:46:12.656828 mkdir("/var/lib/leapp/scratch", 0777) = 0 24271 11:46:12.656885 mkdir("/var/lib/leapp/scratch/mounts", 0777) = 0 24271 11:46:12.776710 mkdir("/var/lib/leapp/scratch/mounts/root_", 0777) = 0 24271 11:46:12.776772 mkdir("/var/lib/leapp/scratch/mounts/root_/upper", 0777) = 0 24271 11:46:12.776866 mkdir("/var/lib/leapp/scratch/mounts/root_/work", 0777) = 0 24271 11:46:12.776954 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay", 0777) = 0 24282 11:46:12.801320 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) 24282 11:46:12.801935 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/lea"...) = 0 24271 11:46:12.930815 mkdir("/var/lib/leapp/scratch/mounts/root_tmp", 0777) = 0 24271 11:46:12.930881 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/upper", 0777) = 0 24271 11:46:12.930973 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/work", 0777) = 0 24271 11:46:12.931066 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", 0777) = 0 24288 11:46:12.956517 mkdir("/run/mount", 0755) = -1 EEXIST (File exists) 24288 11:46:12.957128 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", "overlay", MS_MGC_VAL, "lowerdir=/tmp,upperdir=/var/lib/"...) = 0 24290 11:46:12.998509 newfstatat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", {st_mode=S_IFDIR|0755, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0 24290 11:46:12.998848 unlinkat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", AT_REMOVEDIR) = 0 24271 11:46:13.015461 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", 0777) = -1 EINVAL (Invalid argument)
My current assumption is a problem with overlay and xfs, in very rare conditions.
Attaching the leapp.db they shared.
Version-Release number of selected component (if applicable):
leapp-upgrade-el7toel8-0.17.0-1.el7_9
How reproducible:
Reproducible at will on customer site.
Not able to reproduce internally.
Actual results:
Upgrade is not done.
Additional info:
- A similar (but not identical) issue has been reported in the past (rhbz#1870668).
It has been closed due to insufficient data, and the KCS (#5339021) clearly does not apply for them.
- Workaround was to make Leapp believes it has to create ext4 images as upper layers (as it is done for XFS without ftype)
Edit /usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.py and insert one line (52) to define an array in _prepare_required_mounts():
46 def _prepare_required_mounts(scratch_dir, mounts_dir, mount_points, xfs_info): 47 result = { 48 mount_point.fs_file: mounting.NullMount( 49 _mount_dir(mounts_dir, mount_point.fs_file)) for mount_point in mount_points 50 } 51 52 xfs_info.mountpoints_without_ftype = [ "/", "/var", "/usr", "/boot" ] # <<< 53 if not xfs_info.mountpoints_without_ftype: 54 return result
Additionally, it has been suggested comment out the partitions which are not required during the in-place upgrade and use a larger LEAPP_OVL_SIZE (8192).
- A simple reproducer involving the same set of syscalls has been suggested but the customer didn't spend the time to do so.
yum install strace trace-cmd strace -fTttyyvs 1024 -o /root/reproducer.strace -p $$ & rm -rf /var/lib/leapp/scratch mkdir -p /var/lib/leapp/scratch/mounts/root_/{system_overlay,upper,work} mount -t overlay overlay2 -o lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work /var/lib/leapp/scratch/mounts/root_/system_overlay ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp rm -rf /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp trace-cmd record -o /root/mkdir.trc -b 10000 -F -p function_graph -O hex -- mkdir -m 777 /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay pkill strace umount -fl /var/lib/leapp/scratch/mounts/root_/system_overlay trace-cmd report /root/mkdir.trc > /root/mkdir.trc.txt tar zcf /tmp/debug.tgz /root/mkdir.trc /root/mkdir.trc.txt /root/reproducer.strace /var/lib/leapp/leapp.db
- From mkdir() man page:
EINVAL The final component ("basename") of the new directory's pathname is invalid (e.g., it contains characters not permitted by the underlying filesystem).
- lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 150G 0 disk |-sda1 8:1 0 953M 0 part /boot `-sda2 8:2 0 139.7G 0 part |-rhel-root 253:0 0 23.3G 0 lvm / |-rhel-usr 253:1 0 18.6G 0 lvm /usr |-rhel-tmp 253:2 0 9.3G 0 lvm /tmp |-rhel-swap 253:3 0 9.3G 0 lvm /swap |-rhel-home 253:4 0 28G 0 lvm /home |-rhel-opt 253:5 0 28G 0 lvm /opt `-rhel-var 253:6 0 23.3G 0 lvm /var sr0 11:0 1 1024M 0 rom
- /etc/fstab
/dev/mapper/rhel-root / xfs defaults 0 0 UUID=7a37d1ea-22f6-43b4-bc36-e2dcad4d8212 /boot xfs defaults 0 0 /dev/mapper/rhel-home /home xfs defaults 0 0 /dev/mapper/rhel-opt /opt xfs defaults 0 0 /dev/mapper/rhel-swap /swap xfs defaults 0 0 /dev/mapper/rhel-tmp /tmp xfs defaults 0 0 /dev/mapper/rhel-usr /usr xfs defaults 0 0 /dev/mapper/rhel-var /var xfs defaults 0 0
- blkid
/dev/sda1: UUID="7a37d1ea-22f6-43b4-bc36-e2dcad4d8212" TYPE="xfs" /dev/sda2: UUID="SwTtcI-Zs2a-acFH-MWED-Z1HO-We1q-Ln3JLg" TYPE="LVM2_member" /dev/mapper/rhel-root: UUID="961840e5-dbd0-4112-a243-043073a98092" TYPE="xfs" /dev/mapper/rhel-usr: UUID="a0f82a03-e72a-4109-abad-a0588c1ebf96" TYPE="xfs" /dev/mapper/rhel-tmp: UUID="8d569aed-4522-433f-9f59-2cf02651422b" TYPE="xfs" /dev/mapper/rhel-swap: UUID="25df4b5c-79b3-4453-a993-d1e2df3e3dde" TYPE="xfs" /dev/mapper/rhel-home: UUID="cb4b763e-d4dc-4277-bfb6-7d500b3dafd3" TYPE="xfs" /dev/mapper/rhel-opt: UUID="e547d839-3948-4638-91ca-ab0aba2d1582" TYPE="xfs" /dev/mapper/rhel-var: UUID="6765f39b-b8bc-4b83-814b-f312a522a191" TYPE="xfs"
- grep -o ftype=. sos_commands/xfs/*
sos_commands/xfs/xfs_info:ftype=1 sos_commands/xfs/xfs_info_.boot:ftype=1 sos_commands/xfs/xfs_info_.home:ftype=1 sos_commands/xfs/xfs_info_.opt:ftype=1 sos_commands/xfs/xfs_info_.swap:ftype=1 sos_commands/xfs/xfs_info_.tmp:ftype=1 sos_commands/xfs/xfs_info_.usr:ftype=1 sos_commands/xfs/xfs_info_.var:ftype=1
- default sysctls
- cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-1160.80.1.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/usr rhgb quiet LANG=en_IN.UTF-8