Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-8367

Unable to remove locked or leftover logical volume segment after a pvmove failure.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • rhel-7.9.z
    • lvm2 / Other
    • None
    • Important
    • rhel-sst-logical-storage
    • ssg_filesystems_storage_and_HA
    • 2
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • If docs needed, set a value
    • None

      Description of problem:

      Unable to complete or abort pvmove migration when it failed with error ' Logical volume testvg/pvmove0_mimage_0 in use.'

      1. pvmove --abort
        Device name for LV testvg/pvmove0 not found in metadata. (unfinished pvmove mirror removal?)

      Version-Release number of selected component (if applicable):

      RHEL7.9 /lvm2-2.02.187-6.el7_9.5.x86_64
      issue reproducible with RHEL9.2 as well [lvm2-2.03.17-7.el9.x86_64]

      How reproducible:

      Always

      Steps to Reproduce:

      1. Create volume group and logical volume with multiple PVs.
      2. Start migration using pvmove
      3. Try to access the pvmove image and keep it as busy/open while pvmove is in progress.
      4. After moving all PEs, it failed with the error "LVM command executed by lvmpolld failed."

      Actual results:

      pvmove migration failed and unable to remove the pvmove segments.

      >> Below is my lvm stack and I trying to migrate PEs from sda to sdc.

      1. lvs -ao +devices testvg
        LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices
        lvol0 testvg wi-a---- 6.00g /dev/sda(0)
        lvol0 testvg wi-a---- 6.00g /dev/sdb(0)
        lvol1 testvg wi-a---- 1.00g /dev/sda(1023)

      >> Start migration using pvmove and try to open the device. [ here I used hexdump utility to open the dm-X device].

      1. pvmove --atomic /dev/sda /dev/sdc
        /dev/sda: Moved: 0.00%
        /dev/sda: Moved: 24.63%
        /dev/sda: Moved: 52.85%
        /dev/sda: Moved: 80.30%
        /dev/sda: Moved: 100.00%
        LVM command executed by lvmpolld failed.
        For more information see lvmpolld messages in syslog or lvmpolld log file.

      >> Logical volumes are still accessible and able to mount, but a few pvmove segments still exist, and unable to clear them from lvm stack. Also, the source disk is in use, and unable to remove it from VG.

      1. dmsetup table |grep testvg
        testvg-pvmove0_mimage_1: 0 10477568 error
        testvg-pvmove0_mimage_0: 0 8380416 linear 8:0 2048
        testvg-pvmove0_mimage_0: 8380416 2097152 linear 8:0 8382464
        testvg-lvol1: 0 2097152 linear 253:6 8380416
        testvg-pvmove0: 0 8380416 linear 8:32 2048
        testvg-pvmove0: 8380416 2097152 linear 8:32 8382464
        testvg-lvol0: 0 8380416 linear 253:6 0
        testvg-lvol0: 8380416 4202496 linear 8:16 2048

      >> pvmove abort is failing with below error.

      1. pvmove --abort
        Device name for LV testvg/pvmove0 not found in metadata. (unfinished pvmove mirror removal?)
      1. journalctl -u lvm2-lvmpolld
        • Logs begin at Wed 2023-08-02 12:54:26 EDT, end at Wed 2023-08-02 13:22:17 EDT. –
          Aug 02 13:10:55 node1 systemd[1]: Started LVM2 poll daemon.
          Aug 02 13:11:40 node1 lvmpolld[2501]: W: LVPOLL: PID 2504: STDERR: ' WARNING: This metadata update is NOT backed up.'
          Aug 02 13:14:27 node1 lvmpolld[2501]: W: LVPOLL: PID 2860: STDERR: ' WARNING: This metadata update is NOT backed up.'
          Aug 02 13:16:24 node1 systemd[1]: Started LVM2 poll daemon.
          Aug 02 13:17:24 node1 lvmpolld[3222]: W: LVPOLL: PID 3225: STDERR: ' WARNING: This metadata update is NOT backed up.'
          Aug 02 13:17:29 node1 lvmpolld[3222]: W: LVPOLL: PID 3225: STDERR: ' Logical volume testvg/pvmove0_mimage_0 in use.' <===============
          Aug 02 13:17:29 node1 lvmpolld[3222]: LVMPOLLD: lvm2 cmd (PID 3225) failed (retcode: 5)

      Expected results:

      Looking for some options to clear the leftover pvmove segments properly. There are options to manually edit the VG metadata and restore it back to disk, but for a very large volume, this should be very complicated and may cause the lv in the LOCKED state.

      Additional info:

      This issue is reproducible with RHEL9 standalone systems as well.
      After reproducing this issue, I tried to disable ( use_lvmpolld = 0), but still unable to abort or complete pvmove.

              lvm-team lvm-team
              rhn-support-rmadhuso Ranjith ML
              lvm-team lvm-team
              Cluster QE Cluster QE
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: