-
Bug
-
Resolution: Not a Bug
-
Undefined
-
None
-
rhos-16.2.z
-
None
-
False
-
-
False
-
None
-
-
Description of problem:
Similar to [1] but without the provider_location deduplication, some volumes are attached to different guests using the same mpath block device.
There's a bunch of errors such as those in cinder-volume.log:
~~~
2024-11-29 00:06:19.129 109 ERROR cinder.volume.drivers.hitachi.hbsd.hbsd_utils [req-9312837e-49d5-40ce-b100-93497c480cc4 092614c5c3354831a9cc175848e160b6 71d4e88249104205b08981ef3b4b2321 - default default] MSGID0616-E: A pair cannot be deleted. (P-VOL: 837, S-VOL: 844(PSUE))
2024-11-29 00:06:19.129 109 ERROR cinder.volume.manager [req-9312837e-49d5-40ce-b100-93497c480cc4 092614c5c3354831a9cc175848e160b6 71d4e88249104205b08981ef3b4b2321 - default default] Unable to delete busy volume.: cinder.volume.drivers.hitachi.hbsd.hbsd_exception.HBSDVolumeIsBusy: Volume volume-59947f34-e49b-4571-ab56-e3b532653d1a is busy.
~~~
or:
~~~
2024-11-29 06:33:27.688 109 ERROR oslo_messaging.rpc.server [req-714d22e3-2321-4013-b39d-7c234d7bd86b 092614c5c3354831a9cc175848e160b6 2d7fe62e0f4a4a16b7fbe22f8feef322 - default default] Exception during message handling: cinder.volume.drivers.hitachi.hbsd.hbsd_exception.HBSDError: Bad or unexpected response from the storage volume backend API: The Simple REST API failed. (source: /ConfigurationManager/simple/v1/objects/storages/886000448587/snapshots/2186,3, ID: None, message: An internal error occurred. Wait for a while and try again. If this occurs repeatedly, contact customer support. A snapshot that status is unexpected such as In Sync, Clone Ready or Error may be left. For snap clone or snapshot map operation, an empty volume may be created as a snap clone or mapping target. Delete the snapshot and the volume if necessary, and then retry the operation., cause: None, solution: None, code:
, method: POST, url: https://100.123.26.121:23451/ConfigurationManager/simple/v1/objects/storages/886000448587/snapshots, params: None, body: {'params': [
{'masterVolumeId': 2186, 'poolId': 20, 'snapshotGroupName': 'HBSD-cascade6', 'type': 'Mapped Snapshot', 'destinationSize': 105472}]})
~~~
[1] Partnerhttps://bugzilla.redhat.com/show_bug.cgi?id=2271411
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
Expected results:
Additional info:
cat ansible.out | sed -e "s#.'/dev/disk/by-id/(.)'.*#
1#" | sort | uniq -c | sort -k2,2n
2 dm-uuid-mpath-360050768108101eb000000000000080f
2 dm-uuid-mpath-360050768108101eb0000000000000810
2 dm-uuid-mpath-360050768108101eb0000000000000811
2 dm-uuid-mpath-360050768108101eb0000000000000812
2 dm-uuid-mpath-360050768108101eb0000000000000813
2 dm-uuid-mpath-360060e8012bdcb005040bdcb0000023f
2 dm-uuid-mpath-360060e8012bdcb005040bdcb00000274
2 dm-uuid-mpath-360060e8012bdcb005040bdcb0000029b
2 dm-uuid-mpath-360060e8012bdcb005040bdcb000002c9
2 dm-uuid-mpath-360060e8012bdcb005040bdcb000002ca
2 dm-uuid-mpath-360060e8012bdcb005040bdcb000002cb
2 dm-uuid-mpath-360060e8012bdcb005040bdcb0000032a
2 dm-uuid-mpath-360060e8012bdcb005040bdcb0000034f
2 dm-uuid-mpath-360060e8012bdcb005040bdcb00000367
2 dm-uuid-mpath-360060e8012bdcb005040bdcb00000368
3 dm-uuid-mpath-360060e8012bdcb005040bdcb00000d19
Volume with same dm on compute-0381:
volid: 7f1c45a1 / volid: de7b400f
mpath: 360060e8012bdcb005040bdcb0000032a
different backends for those 2 volumes:
MariaDB [cinder]> select host from volumes where id like '7f1%'\G
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- 1. row ***************************
host: hostgroup@VSPG900_SSD#VSPG900_SSD
1 row in set (0.000 sec)
- 1. row ***************************
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
MariaDB [cinder]> select host from volumes where id like 'de7b%'\G
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- 1. row ***************************
host: hostgroup@VSPG9000#VSPG9000
1 row in set (0.000 sec)
- 1. row ***************************
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
mpath: 360060e8012bdcb005040bdcb0000034f
MariaDB [cinder]> select host from volumes where id like '4894%'\G
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- 1. row ***************************
host: hostgroup@VSPG9000#VSPG9000
1 row in set (0.000 sec)
- 1. row ***************************
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
MariaDB [cinder]> select host from volumes where id like '2af3%'\G
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- 1. row ***************************
host: hostgroup@VSPG9000#VSPG9000
- 1. row ***************************
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
MariaDB [cinder]> select * from volume_attachment where volume_id like '2af3%' and deleted=0\G
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- 1. row ***************************
created_at: 2024-11-26 21:44:56
updated_at: 2024-11-26 21:45:08
deleted_at: NULL
deleted: 0
id: a2d5e93b-9b2a-4b9a-95f5-fff7ab132adc
volume_id: 2af36815-2737-4655-9c46-ebf4a70e2789
attached_host: compute489
instance_uuid: f1e4a49f-423d-44a5-af36-68d4cb71f81c
mountpoint: /dev/vde
attach_time: 2024-11-26 21:44:58
detach_time: NULL
attach_mode: null
attach_status: attached
connection_info: {"target_wwn": ["50060e8012bdcb51", "50060e8012bdcb40", "50060e8012bdcb41", "50060e8012bdcb50"], "target_discovered": false, "access_mode": "rw", "target_lun": 5, "qos_specs": null, "encrypted": false, "driver_volume_type": "fibre_channel", "attachment_id": "a2d5e93b-9b2a-4b9a-95f5-fff7ab132adc"}connector:
{"platform": "x86_64", "os_type": "linux", "ip": "10.8.8.39", "host": "compute489", "multipath": true, "initiator": "iqn.1994-05.com.redhat:d961bdbc1bf4", "wwpns": ["100000620b3b311e", "100000620b3b311f"], "wwnns": ["200000620b3b311e", "200000620b3b311f"], "do_local_attach": false, "system uuid": "21f54d5a-c921-11ed-af3f-72e28429c75b", "mountpoint": "/dev/vde", "mode": null}1 row in set (0.000 sec)
- 1. row ***************************
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
MariaDB [cinder]> select * from volume_attachment where volume_id like '4894%' and deleted=0\G
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- 1. row ***************************
created_at: 2024-11-26 15:39:56
updated_at: 2024-11-26 15:40:00
deleted_at: NULL
deleted: 0
id: d144d291-1a4a-4776-b8d8-e98bf6a20ac1
volume_id: 48942692-1fcc-4cbe-b707-49f29a20bcf5
attached_host: compute0489
instance_uuid: e4fe6bcb-44eb-4125-98eb-a5cad0c05708
mountpoint: /dev/vdc
attach_time: 2024-11-26 15:39:58
detach_time: NULL
attach_mode: rw
attach_status: attached
connection_info: {"target_wwn": ["50060e8012bdcb51", "50060e8012bdcb40", "50060e8012bdcb41", "50060e8012bdcb50"], "target_discovered": false, "access_mode": "rw", "target_lun": 5, "qos_specs": null, "encrypted": false, "driver_volume_type": "fibre_channel", "attachment_id": "d144d291-1a4a-4776-b8d8-e98bf6a20ac1"}connector:
{"platform": "x86_64", "os_type": "linux", "ip": "10.8.8.39", "host": "compute489", "multipath": true, "initiator": "iqn.1994-05.com.redhat:d961bdbc1bf4", "wwpns": ["100000620b3b311e", "100000620b3b311f"], "wwnns": ["200000620b3b311e", "200000620b3b311f"], "do_local_attach": false, "system uuid": "21f54d5a-c921-11ed-af3f-72e28429c75b", "mountpoint": "/dev/vdc"}
- 1. row ***************************
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- external trackers