-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
rhel-10.2
-
None
-
No
-
None
-
rhel-storage-lvm
-
None
-
False
-
False
-
-
None
-
None
-
None
-
None
-
Unspecified
-
Unspecified
-
Unspecified
-
-
x86_64
-
None
I need to go though prior releases and see where this was introduced. This is likely a regression.
This scenario was developed for raid issues https://issues.redhat.com/browse/RHEL-89832 and https://issues.redhat.com/browse/RHEL-67039 and recently added to run on mirror volumes as well as a variety of other lvm non raid types. This scenario no longer passes in the latest build. I was also able to reproduce this in a 9.7 build as well.
kernel-6.12.0-126.el10 BUILT: Thu Sep 4 05:53:31 PM CEST 2025
lvm2-2.03.35-1.el10 BUILT: Wed Sep 10 05:00:31 PM CEST 2025
lvm2-libs-2.03.35-1.el10 BUILT: Wed Sep 10 05:00:31 PM CEST 2025
SCENARIO - activation_after_transient_subset_device_removals: Test how LVM and devicesfile reacts to transient removals of a subset of devices for different segtypes (RHEL-67039|RHEL-89832)
vgcreate devfile_VG /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
Volume group "devfile_VG" successfully created
[...]
------------------------------------------------------------
lvcreate --yes --type mirror -n devfile_LV -l100%VG -m 3 devfile_VG
Logical volume "devfile_LV" created.
devfile_LV devfile_VG mwi-a-m--- <35.00g [devfile_LV_mlog] 1.02 devfile_LV_mimage_0(0),devfile_LV_mimage_1(0),devfile_LV_mimage_2(0),devfile_LV_mimage_3(0) mirror
[devfile_LV_mimage_0] devfile_VG Iwi-aom--- <35.00g /dev/sda(0) linear
[devfile_LV_mimage_1] devfile_VG Iwi-aom--- <35.00g /dev/sdb(0) linear
[devfile_LV_mimage_2] devfile_VG Iwi-aom--- <35.00g /dev/sdc(0) linear
[devfile_LV_mimage_3] devfile_VG Iwi-aom--- <35.00g /dev/sdd(0) linear
[devfile_LV_mlog] devfile_VG lwi-aom--- 4.00m /dev/sdf(0) linear
mkfs --type xfs -f /dev/devfile_VG/devfile_LV
mount /dev/devfile_VG/devfile_LV /mnt/devfile_LV
Writing files to /mnt/devfile_LV
/usr/tests/sts-rhel10.0/bin/checkit -w /mnt/devfile_LV -f /tmp/Filesystem.1266217 -n 4500
checkit starting with:
CREATE
/usr/tests/sts-rhel10.0/bin/checkit -w /mnt/devfile_LV -f /tmp/Filesystem.1266217 -v
checkit starting with:
VERIFY
Disabling device sdb on virt-497.cluster-qe.lab.eng.brq.redhat.com
'echo offline > /sys/block/sdb/device/state'
Disabling device sda on virt-497.cluster-qe.lab.eng.brq.redhat.com
'echo offline > /sys/block/sda/device/state'
Disabling device sdd on virt-497.cluster-qe.lab.eng.brq.redhat.com
'echo offline > /sys/block/sdd/device/state'
Sep 23 16:38:38 virt-497 kernel: sd 5:0:0:0: rejecting I/O to offline device
Sep 23 16:38:38 virt-497 dmeventd[255123]: Secondary mirror device 253:6 flush failed.
Sep 23 16:38:38 virt-497 dmeventd[255123]: Device failure in devfile_VG-devfile_LV.
Sep 23 16:38:38 virt-497 kernel: recovery_complete: 2 callbacks suppressed
Sep 23 16:38:38 virt-497 kernel: device-mapper: raid1: Write error during recovery (error = 0x1)
Sep 23 16:38:38 virt-497 kernel: device-mapper: raid1: Write error during recovery (error = 0x1)
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid uy9P2h-9KaF-Yf6g-YvMi-SrHY-iypF-V96tdn.
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid JWeTbA-hrMl-N29o-hjFe-7hkx-Tb42-as1fBp.
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid gYq967-RHRk-Lu1A-w3RW-O9DH-cnwD-r1jtZK.
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: VG devfile_VG is missing PV uy9P2h-9KaF-Yf6g-YvMi-SrHY-iypF-V96tdn (last written to [unknown]).
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: VG devfile_VG is missing PV JWeTbA-hrMl-N29o-hjFe-7hkx-Tb42-as1fBp (last written to [unknown]).
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: VG devfile_VG is missing PV gYq967-RHRk-Lu1A-w3RW-O9DH-cnwD-r1jtZK (last written to /dev/sdd).
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid uy9P2h-9KaF-Yf6g-YvMi-SrHY-iypF-V96tdn.
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid JWeTbA-hrMl-N29o-hjFe-7hkx-Tb42-as1fBp.
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid gYq967-RHRk-Lu1A-w3RW-O9DH-cnwD-r1jtZK.
Sep 23 16:38:38 virt-497 kernel: device-mapper: raid1: Write error during recovery (error = 0x1)
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid uy9P2h-9KaF-Yf6g-YvMi-SrHY-iypF-V96tdn.
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid JWeTbA-hrMl-N29o-hjFe-7hkx-Tb42-as1fBp.
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid gYq967-RHRk-Lu1A-w3RW-O9DH-cnwD-r1jtZK.
Sep 23 16:38:38 virt-497 kernel: device-mapper: raid1: Write error during recovery (error = 0x1)
Sep 23 16:38:38 virt-497 kernel: device-mapper: raid1: Write error during recovery (error = 0x1)
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid uy9P2h-9KaF-Yf6g-YvMi-SrHY-iypF-V96tdn.
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid JWeTbA-hrMl-N29o-hjFe-7hkx-Tb42-as1fBp.
Sep 23 16:38:38 virt-497 dmeventd[255123]: WARNING: Couldn't find device with uuid gYq967-RHRk-Lu1A-w3RW-O9DH-cnwD-r1jtZK.
Sep 23 16:38:38 virt-497 dmeventd[255123]: Mirror status: 1 of 2 images failed.
Sep 23 16:38:38 virt-497 dmeventd[255123]: Repair of mirrored device devfile_VG-devfile_LV finished successfully.
Sep 23 16:38:38 virt-497 dmeventd[255123]: devfile_VG-devfile_LV has unmirrored portion.
Sep 23 16:38:39 virt-497 qarshd[255690]: Running cmdline: lvs -a -o +devices,segtype | grep devfile_LV
devfile_LV devfile_VG -wi-ao---- <35.00g /dev/sdc(0) linear
WARNING: Couldn't find device with uuid uy9P2h-9KaF-Yf6g-YvMi-SrHY-iypF-V96tdn.
WARNING: Couldn't find device with uuid JWeTbA-hrMl-N29o-hjFe-7hkx-Tb42-as1fBp.
WARNING: Couldn't find device with uuid gYq967-RHRk-Lu1A-w3RW-O9DH-cnwD-r1jtZK.
WARNING: VG devfile_VG is missing PV uy9P2h-9KaF-Yf6g-YvMi-SrHY-iypF-V96tdn (last written to [unknown]).
WARNING: VG devfile_VG is missing PV JWeTbA-hrMl-N29o-hjFe-7hkx-Tb42-as1fBp (last written to [unknown]).
WARNING: VG devfile_VG is missing PV gYq967-RHRk-Lu1A-w3RW-O9DH-cnwD-r1jtZK (last written to [unknown]).
umount /mnt/devfile_LV
deactivating all LVs in devfile_VG on virt-497.cluster-qe.lab.eng.brq.redhat.com
vgchange --yes -an devfile_VG
Enabling device sdb on virt-497.cluster-qe.lab.eng.brq.redhat.com
'echo running > /sys/block/sdb/device/state'
Enabling device sda on virt-497.cluster-qe.lab.eng.brq.redhat.com
'echo running > /sys/block/sda/device/state'
Enabling device sdd on virt-497.cluster-qe.lab.eng.brq.redhat.com
'echo running > /sys/block/sdd/device/state'
devfile_LV devfile_VG -wi------- <35.00g /dev/sdc(0) linear
WARNING: Ignoring metadata seqno 5 on /dev/sda for seqno 10 on /dev/sdc for VG devfile_VG.
WARNING: Ignoring metadata seqno 8 on /dev/sdd for seqno 10 on /dev/sdc for VG devfile_VG.
WARNING: Inconsistent metadata found for VG devfile_VG.
See vgck --updatemetadata to correct inconsistency.
WARNING: VG devfile_VG was previously updated while PV /dev/sda was missing.
WARNING: VG devfile_VG was previously updated while PV /dev/sdb was missing.
WARNING: VG devfile_VG was previously updated while PV /dev/sdd was missing.
WARNING: VG devfile_VG has unused reappeared PV /dev/sda uy9P2h-9KaF-Yf6g-YvMi-SrHY-iypF-V96tdn.
WARNING: VG devfile_VG has unused reappeared PV /dev/sdb JWeTbA-hrMl-N29o-hjFe-7hkx-Tb42-as1fBp.
WARNING: VG devfile_VG has unused reappeared PV /dev/sdd gYq967-RHRk-Lu1A-w3RW-O9DH-cnwD-r1jtZK.
activating all LVs in devfile_VG on virt-497.cluster-qe.lab.eng.brq.redhat.com
vgchange --yes -ay devfile_VG
[root@virt-497 ~]# mount /dev/devfile_VG/devfile_LV /mnt/devfile_LV
mount: /mnt/devfile_LV: can't read superblock on /dev/mapper/devfile_VG-devfile_LV.
dmesg(1) may have more information after failed mount system call.
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): Mounting V5 Filesystem 638dcb27-fce7-4ca5-ae08-70965148ec88
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): Metadata CRC error detected at xfs_agf_read_verify+0x7b/0x110 [xfs], xfs_agf block 0x347e801
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): Unmount and run xfs_repair
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): First 128 bytes of corrupted metadata buffer:
Sep 23 17:46:27 virt-497 kernel: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Sep 23 17:46:27 virt-497 kernel: 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Sep 23 17:46:27 virt-497 kernel: 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Sep 23 17:46:27 virt-497 kernel: 00000030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Sep 23 17:46:27 virt-497 kernel: 00000040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Sep 23 17:46:27 virt-497 kernel: 00000050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Sep 23 17:46:27 virt-497 kernel: 00000060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Sep 23 17:46:27 virt-497 kernel: 00000070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): metadata I/O error in "xfs_read_agf+0x97/0x150 [xfs]" at daddr 0x347e801 len 1 error 74
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): Error -117 reserving per-AG metadata reserve pool.
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): Corruption of in-memory data (0x8) detected at xfs_fs_reserve_ag_blocks+0xbf/0xf0 [xfs] (fs/xfs/xfs_fsops.c:552). Shutting down filesystem.
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): Please unmount the filesystem and rectify the problem(s)
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): Ending clean mount
Sep 23 17:46:27 virt-497 kernel: XFS (dm-2): Error -5 reserving per-AG metadata reserve pool.
