Description of problem:
Leapp failed at the preupgrade step due to a directory that cannot be created on top of <overlayfs>/tmp.
The issue was not with this specific partition, if this one is not used it fails on the next one.

• They use the latest el7 kernel and the latest leapp packages
• They have no 3rd party tainting the kernel.
• All their XFS have an ftype (hence it supposedly supports overlays).
• The strace shows the first mkdir() on the mounted overlay fails with an EINVAL errno (excerpt of relevant calls):

24271 11:46:12.656828 mkdir("/var/lib/leapp/scratch", 0777) = 0
24271 11:46:12.656885 mkdir("/var/lib/leapp/scratch/mounts", 0777) = 0
24271 11:46:12.776710 mkdir("/var/lib/leapp/scratch/mounts/root_", 0777) = 0
24271 11:46:12.776772 mkdir("/var/lib/leapp/scratch/mounts/root_/upper", 0777) = 0
24271 11:46:12.776866 mkdir("/var/lib/leapp/scratch/mounts/root_/work", 0777) = 0
24271 11:46:12.776954 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay", 0777) = 0
24282 11:46:12.801320 mkdir("/run/mount", 0755) = -1 EEXIST (File exists)
24282 11:46:12.801935 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/lea"...) = 0
24271 11:46:12.930815 mkdir("/var/lib/leapp/scratch/mounts/root_tmp", 0777) = 0
24271 11:46:12.930881 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/upper", 0777) = 0
24271 11:46:12.930973 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/work", 0777) = 0
24271 11:46:12.931066 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", 0777) = 0
24288 11:46:12.956517 mkdir("/run/mount", 0755) = -1 EEXIST (File exists)
24288 11:46:12.957128 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", "overlay", MS_MGC_VAL, "lowerdir=/tmp,upperdir=/var/lib/"...) = 0
24290 11:46:12.998509 newfstatat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", {st_mode=S_IFDIR|0755, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
24290 11:46:12.998848 unlinkat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", AT_REMOVEDIR) = 0
24271 11:46:13.015461 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", 0777) = -1 EINVAL (Invalid argument)

My current assumption is a problem with overlay and xfs, in very rare conditions.
Attaching the leapp.db they shared.

Version-Release number of selected component (if applicable):
leapp-upgrade-el7toel8-0.17.0-1.el7_9

How reproducible:
Reproducible at will on customer site.
Not able to reproduce internally.

Actual results:

Upgrade is not done.

Additional info:

• A similar (but not identical) issue has been reported in the past (rhbz#1870668).
  It has been closed due to insufficient data, and the KCS (#5339021) clearly does not apply for them.

• Workaround was to make Leapp believes it has to create ext4 images as upper layers (as it is done for XFS without ftype)

Edit /usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.py and insert one line (52) to define an array in _prepare_required_mounts():

 46 def _prepare_required_mounts(scratch_dir, mounts_dir, mount_points, xfs_info):
 47     result = {
 48         mount_point.fs_file: mounting.NullMount(
 49             _mount_dir(mounts_dir, mount_point.fs_file)) for mount_point in mount_points
 50     }
 51 
 52     xfs_info.mountpoints_without_ftype = [ "/", "/var", "/usr", "/boot" ]  # <<<      
 53     if not xfs_info.mountpoints_without_ftype:
 54         return result

Additionally, it has been suggested comment out the partitions which are not required during the in-place upgrade and use a larger LEAPP_OVL_SIZE (8192).

• A simple reproducer involving the same set of syscalls has been suggested but the customer didn't spend the time to do so.

  yum install strace trace-cmd
  strace -fTttyyvs 1024 -o /root/reproducer.strace -p $$ &
  rm -rf /var/lib/leapp/scratch
  mkdir -p /var/lib/leapp/scratch/mounts/root_/{system_overlay,upper,work}
  mount -t overlay overlay2 -o lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work /var/lib/leapp/scratch/mounts/root_/system_overlay
  ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay
  ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp
  rm -rf /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp
  trace-cmd record -o /root/mkdir.trc -b 10000 -F -p function_graph -O hex -- mkdir -m 777 /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp
  ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay
  pkill strace
  umount -fl /var/lib/leapp/scratch/mounts/root_/system_overlay
  trace-cmd report /root/mkdir.trc > /root/mkdir.trc.txt
  tar zcf /tmp/debug.tgz /root/mkdir.trc /root/mkdir.trc.txt /root/reproducer.strace /var/lib/leapp/leapp.db
  

• From mkdir() man page:

         EINVAL The final component ("basename") of the new directory's pathname is invalid 
                (e.g., it contains characters not permitted by the underlying filesystem).
  

• lsblk

  NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
  sda             8:0    0   150G  0 disk 
  |-sda1          8:1    0   953M  0 part /boot
  `-sda2          8:2    0 139.7G  0 part 
    |-rhel-root 253:0    0  23.3G  0 lvm  /
    |-rhel-usr  253:1    0  18.6G  0 lvm  /usr
    |-rhel-tmp  253:2    0   9.3G  0 lvm  /tmp
    |-rhel-swap 253:3    0   9.3G  0 lvm  /swap
    |-rhel-home 253:4    0    28G  0 lvm  /home
    |-rhel-opt  253:5    0    28G  0 lvm  /opt
    `-rhel-var  253:6    0  23.3G  0 lvm  /var
  sr0            11:0    1  1024M  0 rom  
  

• /etc/fstab

  /dev/mapper/rhel-root   /                       xfs     defaults        0 0
  UUID=7a37d1ea-22f6-43b4-bc36-e2dcad4d8212 /boot                   xfs     defaults        0 0
  /dev/mapper/rhel-home   /home                   xfs     defaults        0 0
  /dev/mapper/rhel-opt    /opt                    xfs     defaults        0 0
  /dev/mapper/rhel-swap   /swap                   xfs     defaults        0 0
  /dev/mapper/rhel-tmp    /tmp                    xfs     defaults        0 0
  /dev/mapper/rhel-usr    /usr                    xfs     defaults        0 0
  /dev/mapper/rhel-var    /var                    xfs     defaults        0 0
  

• blkid

  /dev/sda1: UUID="7a37d1ea-22f6-43b4-bc36-e2dcad4d8212" TYPE="xfs" 
  /dev/sda2: UUID="SwTtcI-Zs2a-acFH-MWED-Z1HO-We1q-Ln3JLg" TYPE="LVM2_member" 
  /dev/mapper/rhel-root: UUID="961840e5-dbd0-4112-a243-043073a98092" TYPE="xfs" 
  /dev/mapper/rhel-usr: UUID="a0f82a03-e72a-4109-abad-a0588c1ebf96" TYPE="xfs" 
  /dev/mapper/rhel-tmp: UUID="8d569aed-4522-433f-9f59-2cf02651422b" TYPE="xfs" 
  /dev/mapper/rhel-swap: UUID="25df4b5c-79b3-4453-a993-d1e2df3e3dde" TYPE="xfs" 
  /dev/mapper/rhel-home: UUID="cb4b763e-d4dc-4277-bfb6-7d500b3dafd3" TYPE="xfs" 
  /dev/mapper/rhel-opt: UUID="e547d839-3948-4638-91ca-ab0aba2d1582" TYPE="xfs" 
  /dev/mapper/rhel-var: UUID="6765f39b-b8bc-4b83-814b-f312a522a191" TYPE="xfs" 
  

• grep -o ftype=. sos_commands/xfs/*

  sos_commands/xfs/xfs_info:ftype=1
  sos_commands/xfs/xfs_info_.boot:ftype=1
  sos_commands/xfs/xfs_info_.home:ftype=1
  sos_commands/xfs/xfs_info_.opt:ftype=1
  sos_commands/xfs/xfs_info_.swap:ftype=1
  sos_commands/xfs/xfs_info_.tmp:ftype=1
  sos_commands/xfs/xfs_info_.usr:ftype=1
  sos_commands/xfs/xfs_info_.var:ftype=1
  

• default sysctls

• cmdline

  BOOT_IMAGE=/vmlinuz-3.10.0-1160.80.1.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/usr rhgb quiet LANG=en_IN.UTF-8