Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-3320

[storage] Leapp can fail when there are too many LV partitions

Details

    • Major
    • sst_upgrades
    • False
    • Hide

      None

      Show
      None
    • If docs needed, set a value

    Description

      Description of problem:
      Leapp crashes when there are too many LV partitions.
      Customer had at least 55 partitions which were not useful for the IPU.
      Commenting them out from fstab allows to go ahead.

      Version-Release number of selected component (if applicable):
      leapp-upgrade-el7toel8-0.17.0-1.el7_9.noarch

      How reproducible:
      Always

      Steps to Reproduce:
      1/ create a sparse file, it will be used as a temporary block device (here of 20GB)

      1. dd if=/dev/zero of=/root/block seek=40M count=1

      2/ create 100 LV of 100MB under the VG "test".

      1. losetup /dev/loop0 /root/block
      2. pvcreate /dev/loop0
      3. vgcreate test /dev/loop0
      4. for i in $(seq 1 100); do lvcreate -L 100M -n lv$i test; done
      5. vgchange -ay test

      3/ format them (here as ext3 like the customer, but I guess other fstype give the same symptoms)

      1. for i in $(seq 1 100); do mkfs.ext3 -F /dev/test/lv$i; done
      2. for i in $(seq 1 100); do mkdir /srv/fs$i; done
      3. for i in $(seq 1 100); do echo "/dev/test/lv$i /srv/fs$i ext3 defaults 0 0" >> /etc/fstab; done
      4. mount -a

      4/ run a `leapp preupgrade`

      Actual results:
      2022-11-16 05:48:15.525 DEBUG PID: 24941 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['mount', '-t', 'overlay', 'overlay2', '-o', 'lowerdir=/srv/fs45,upperdir=/var/lib/leapp/scratch/mounts/root_srv_fs45/upper,workdir=/var/lib/leapp/scratch/mounts/root_srv_fs45/work', '/var/lib/leapp/scratch/mounts/root_srv_fs45/root_srv_fs45']
      2022-11-16 05:48:15.544 DEBUG PID: 24941 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['mount', '-t', 'overlay', 'overlay2', '-o', 'lowerdir=/srv/fs45,upperdir=/var/lib/leapp/scratch/mounts/root_srv_fs45/upper,workdir=/var/lib/leapp/scratch/mounts/root_srv_fs45/work', '/var/lib/leapp/scratch/mounts/root_srv_fs45/root_srv_fs45']
      2022-11-16 05:48:15.549 DEBUG PID: 24941 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['rm', '-rf', u'/var/lib/leapp/scratch/mounts/root_/system_overlay/srv/fs45']
      Process Process-464:
      Traceback (most recent call last):
      File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      self.run()
      File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
      self._target(*self._args, **self._kwargs)
      File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 72, in _do_run
      actor_instance.run(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/leapp/actors/_init_.py", line 289, in run
      self.process(*args)
      File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 52, in process
      userspacegen.perform()
      File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
      return target_item(*args, **kwargs)
      File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 671, in perform
      xfs_info=indata.xfs_info) as overlay:
      File "/usr/lib64/python2.7/contextlib.py", line 17, in _enter_
      return self.gen.next()
      File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.py", line 229, in create_source_overlay
      cleanup_scratch(scratch_dir, mounts_dir)
      File "/usr/share/leapp-repository/repositories/system_upgrade/common/libraries/overlaygen.py", line 118, in cleanup_scratch
      api.current_logger().debug('Cleaning up mounts')
      File "/usr/lib64/python2.7/logging/_init_.py", line 1137, in debug
      self._log(DEBUG, msg, args, **kwargs)
      File "/usr/lib64/python2.7/logging/_init_.py", line 1268, in _log
      self.handle(record)
      File "/usr/lib64/python2.7/logging/_init_.py", line 1278, in handle
      self.callHandlers(record)
      File "/usr/lib64/python2.7/logging/_init_.py", line 1318, in callHandlers
      hdlr.handle(record)
      File "/usr/lib64/python2.7/logging/_init_.py", line 749, in handle
      self.emit(record)
      File "/usr/lib/python2.7/site-packages/leapp/logger/_init_.py", line 40, in emit
      self._do_emit(log_data)
      File "/usr/lib/python2.7/site-packages/leapp/logger/_init_.py", line 45, in _do_emit
      Audit(**log_data).store()
      File "/usr/lib/python2.7/site-packages/leapp/utils/audit/_init_.py", line 87, in store
      with get_connection(db) as connection:
      File "/usr/lib/python2.7/site-packages/leapp/utils/audit/_init_.py", line 73, in get_connection
      return create_connection(cfg.get('database', 'path'))
      File "/usr/lib/python2.7/site-packages/leapp/cli/commands/upgrade/util.py", line 26, in wrapper
      return f(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/leapp/utils/audit/_init_.py", line 60, in create_connection
      return _initialize_database(sqlite3.connect(path))
      OperationalError: unable to open database file

      =========================================================================================================
      Actor target_userspace_creator unexpectedly terminated with exit code: 1 - Please check the above details

      Expected results:
      No crash.
      Maybe an inhibitor telling to comment out partitions which are definitely not required during the IPU?
      (that would also help in the XFS case where many ext4 images are created to circumvent an old xfs issue)

      Additional info:
      After the crash, partitions in /var/lib/leapp/scratch are still there.

      To clean up the system, lazy-unmount those overlays:

      1. for mp in `mount | awk '/leapp.scratch/ {print $1}

        '`; do umount -vl $mp; done

      2. rm -rf /var/lib/leapp/*

      Attachments

        Activity

          People

            leapp-notifications leapp-notifications
            rhn-support-cbesson Christophe Besson
            leapp-notifications leapp-notifications
            RHEL Upgrades QE Team RHEL Upgrades QE Team
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: