Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-3330

[leapp] IPU 7>8: EINVAL returned by mkdir() on overlayfs with XFS

    • Minor
    • sst_upgrades
    • 3
    • False
    • Hide

      None

      Show
      None
    • None
    • If docs needed, set a value
    • None

      Description of problem:
      Leapp failed at the preupgrade step due to a directory that cannot be created on top of <overlayfs>/tmp.
      The issue was not with this specific partition, if this one is not used it fails on the next one.

      • They use the latest el7 kernel and the latest leapp packages
      • They have no 3rd party tainting the kernel.
      • All their XFS have an ftype (hence it supposedly supports overlays).
      • The strace shows the first mkdir() on the mounted overlay fails with an EINVAL errno (excerpt of relevant calls):
      24271 11:46:12.656828 mkdir("/var/lib/leapp/scratch", 0777) = 0
      24271 11:46:12.656885 mkdir("/var/lib/leapp/scratch/mounts", 0777) = 0
      24271 11:46:12.776710 mkdir("/var/lib/leapp/scratch/mounts/root_", 0777) = 0
      24271 11:46:12.776772 mkdir("/var/lib/leapp/scratch/mounts/root_/upper", 0777) = 0
      24271 11:46:12.776866 mkdir("/var/lib/leapp/scratch/mounts/root_/work", 0777) = 0
      24271 11:46:12.776954 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay", 0777) = 0
      24282 11:46:12.801320 mkdir("/run/mount", 0755) = -1 EEXIST (File exists)
      24282 11:46:12.801935 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/lea"...) = 0
      24271 11:46:12.930815 mkdir("/var/lib/leapp/scratch/mounts/root_tmp", 0777) = 0
      24271 11:46:12.930881 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/upper", 0777) = 0
      24271 11:46:12.930973 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/work", 0777) = 0
      24271 11:46:12.931066 mkdir("/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", 0777) = 0
      24288 11:46:12.956517 mkdir("/run/mount", 0755) = -1 EEXIST (File exists)
      24288 11:46:12.957128 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_tmp/root_tmp", "overlay", MS_MGC_VAL, "lowerdir=/tmp,upperdir=/var/lib/"...) = 0
      24290 11:46:12.998509 newfstatat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", {st_mode=S_IFDIR|0755, st_size=6, ...}, AT_SYMLINK_NOFOLLOW) = 0
      24290 11:46:12.998848 unlinkat(AT_FDCWD, "/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", AT_REMOVEDIR) = 0
      24271 11:46:13.015461 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/tmp", 0777) = -1 EINVAL (Invalid argument)
      

      My current assumption is a problem with overlay and xfs, in very rare conditions.
      Attaching the leapp.db they shared.

      Version-Release number of selected component (if applicable):
      leapp-upgrade-el7toel8-0.17.0-1.el7_9

      How reproducible:
      Reproducible at will on customer site.
      Not able to reproduce internally.

      Actual results:

      Upgrade is not done.

      Additional info:

      • A similar (but not identical) issue has been reported in the past (rhbz#1870668).
        It has been closed due to insufficient data, and the KCS (#5339021) clearly does not apply for them.
      • Workaround was to make Leapp believes it has to create ext4 images as upper layers (as it is done for XFS without ftype)

      Edit /usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.py and insert one line (52) to define an array in _prepare_required_mounts():

       46 def _prepare_required_mounts(scratch_dir, mounts_dir, mount_points, xfs_info):
       47     result = {
       48         mount_point.fs_file: mounting.NullMount(
       49             _mount_dir(mounts_dir, mount_point.fs_file)) for mount_point in mount_points
       50     }
       51 
       52     xfs_info.mountpoints_without_ftype = [ "/", "/var", "/usr", "/boot" ]  # <<<      
       53     if not xfs_info.mountpoints_without_ftype:
       54         return result
      

      Additionally, it has been suggested comment out the partitions which are not required during the in-place upgrade and use a larger LEAPP_OVL_SIZE (8192).

      • A simple reproducer involving the same set of syscalls has been suggested but the customer didn't spend the time to do so.
        yum install strace trace-cmd
        strace -fTttyyvs 1024 -o /root/reproducer.strace -p $$ &
        rm -rf /var/lib/leapp/scratch
        mkdir -p /var/lib/leapp/scratch/mounts/root_/{system_overlay,upper,work}
        mount -t overlay overlay2 -o lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work /var/lib/leapp/scratch/mounts/root_/system_overlay
        ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay
        ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp
        rm -rf /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp
        trace-cmd record -o /root/mkdir.trc -b 10000 -F -p function_graph -O hex -- mkdir -m 777 /var/lib/leapp/scratch/mounts/root_/system_overlay/tmp
        ls -l /var/lib/leapp/scratch/mounts/root_/system_overlay
        pkill strace
        umount -fl /var/lib/leapp/scratch/mounts/root_/system_overlay
        trace-cmd report /root/mkdir.trc > /root/mkdir.trc.txt
        tar zcf /tmp/debug.tgz /root/mkdir.trc /root/mkdir.trc.txt /root/reproducer.strace /var/lib/leapp/leapp.db
        
      • From mkdir() man page:
               EINVAL The final component ("basename") of the new directory's pathname is invalid 
                      (e.g., it contains characters not permitted by the underlying filesystem).
        
      • lsblk
        NAME          MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
        sda             8:0    0   150G  0 disk 
        |-sda1          8:1    0   953M  0 part /boot
        `-sda2          8:2    0 139.7G  0 part 
          |-rhel-root 253:0    0  23.3G  0 lvm  /
          |-rhel-usr  253:1    0  18.6G  0 lvm  /usr
          |-rhel-tmp  253:2    0   9.3G  0 lvm  /tmp
          |-rhel-swap 253:3    0   9.3G  0 lvm  /swap
          |-rhel-home 253:4    0    28G  0 lvm  /home
          |-rhel-opt  253:5    0    28G  0 lvm  /opt
          `-rhel-var  253:6    0  23.3G  0 lvm  /var
        sr0            11:0    1  1024M  0 rom  
        
      • /etc/fstab
        /dev/mapper/rhel-root   /                       xfs     defaults        0 0
        UUID=7a37d1ea-22f6-43b4-bc36-e2dcad4d8212 /boot                   xfs     defaults        0 0
        /dev/mapper/rhel-home   /home                   xfs     defaults        0 0
        /dev/mapper/rhel-opt    /opt                    xfs     defaults        0 0
        /dev/mapper/rhel-swap   /swap                   xfs     defaults        0 0
        /dev/mapper/rhel-tmp    /tmp                    xfs     defaults        0 0
        /dev/mapper/rhel-usr    /usr                    xfs     defaults        0 0
        /dev/mapper/rhel-var    /var                    xfs     defaults        0 0
        
      • blkid
        /dev/sda1: UUID="7a37d1ea-22f6-43b4-bc36-e2dcad4d8212" TYPE="xfs" 
        /dev/sda2: UUID="SwTtcI-Zs2a-acFH-MWED-Z1HO-We1q-Ln3JLg" TYPE="LVM2_member" 
        /dev/mapper/rhel-root: UUID="961840e5-dbd0-4112-a243-043073a98092" TYPE="xfs" 
        /dev/mapper/rhel-usr: UUID="a0f82a03-e72a-4109-abad-a0588c1ebf96" TYPE="xfs" 
        /dev/mapper/rhel-tmp: UUID="8d569aed-4522-433f-9f59-2cf02651422b" TYPE="xfs" 
        /dev/mapper/rhel-swap: UUID="25df4b5c-79b3-4453-a993-d1e2df3e3dde" TYPE="xfs" 
        /dev/mapper/rhel-home: UUID="cb4b763e-d4dc-4277-bfb6-7d500b3dafd3" TYPE="xfs" 
        /dev/mapper/rhel-opt: UUID="e547d839-3948-4638-91ca-ab0aba2d1582" TYPE="xfs" 
        /dev/mapper/rhel-var: UUID="6765f39b-b8bc-4b83-814b-f312a522a191" TYPE="xfs" 
        
      • grep -o ftype=. sos_commands/xfs/*
        sos_commands/xfs/xfs_info:ftype=1
        sos_commands/xfs/xfs_info_.boot:ftype=1
        sos_commands/xfs/xfs_info_.home:ftype=1
        sos_commands/xfs/xfs_info_.opt:ftype=1
        sos_commands/xfs/xfs_info_.swap:ftype=1
        sos_commands/xfs/xfs_info_.tmp:ftype=1
        sos_commands/xfs/xfs_info_.usr:ftype=1
        sos_commands/xfs/xfs_info_.var:ftype=1
        
      • default sysctls
      • cmdline
        BOOT_IMAGE=/vmlinuz-3.10.0-1160.80.1.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/usr rhgb quiet LANG=en_IN.UTF-8
        

            leapp-notifications leapp-notifications
            rhn-support-cbesson Christophe Besson
            leapp-notifications leapp-notifications
            RHEL Upgrades QE Team RHEL Upgrades QE Team
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: