Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-13292

BZ#2332297 on hitachi bakend, some volumes are attached to different guests using the same mpath block device.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • rhos-16.2.z
    • openstack-cinder
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • None

      Description of problem:
      Similar to [1] but without the provider_location deduplication, some volumes are attached to different guests using the same mpath block device.

      There's a bunch of errors such as those in cinder-volume.log:
      ~~~
      2024-11-29 00:06:19.129 109 ERROR cinder.volume.drivers.hitachi.hbsd.hbsd_utils [req-9312837e-49d5-40ce-b100-93497c480cc4 092614c5c3354831a9cc175848e160b6 71d4e88249104205b08981ef3b4b2321 - default default] MSGID0616-E: A pair cannot be deleted. (P-VOL: 837, S-VOL: 844(PSUE))
      2024-11-29 00:06:19.129 109 ERROR cinder.volume.manager [req-9312837e-49d5-40ce-b100-93497c480cc4 092614c5c3354831a9cc175848e160b6 71d4e88249104205b08981ef3b4b2321 - default default] Unable to delete busy volume.: cinder.volume.drivers.hitachi.hbsd.hbsd_exception.HBSDVolumeIsBusy: Volume volume-59947f34-e49b-4571-ab56-e3b532653d1a is busy.
      ~~~
      or:
      ~~~
      2024-11-29 06:33:27.688 109 ERROR oslo_messaging.rpc.server [req-714d22e3-2321-4013-b39d-7c234d7bd86b 092614c5c3354831a9cc175848e160b6 2d7fe62e0f4a4a16b7fbe22f8feef322 - default default] Exception during message handling: cinder.volume.drivers.hitachi.hbsd.hbsd_exception.HBSDError: Bad or unexpected response from the storage volume backend API: The Simple REST API failed. (source: /ConfigurationManager/simple/v1/objects/storages/886000448587/snapshots/2186,3, ID: None, message: An internal error occurred. Wait for a while and try again. If this occurs repeatedly, contact customer support. A snapshot that status is unexpected such as In Sync, Clone Ready or Error may be left. For snap clone or snapshot map operation, an empty volume may be created as a snap clone or mapping target. Delete the snapshot and the volume if necessary, and then retry the operation., cause: None, solution: None, code:

      {'SSB1': '2EC6', 'SSB2': 'FEEC'}

      , method: POST, url: https://100.123.26.121:23451/ConfigurationManager/simple/v1/objects/storages/886000448587/snapshots, params: None, body: {'params': [

      {'masterVolumeId': 2186, 'poolId': 20, 'snapshotGroupName': 'HBSD-cascade6', 'type': 'Mapped Snapshot', 'destinationSize': 105472}

      ]})
      ~~~

      [1] Partnerhttps://bugzilla.redhat.com/show_bug.cgi?id=2271411

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:
      1.
      2.
      3.

      Actual results:

      Expected results:

      Additional info:

      cat ansible.out | sed -e "s#.'/dev/disk/by-id/(.)'.*#
      1#" | sort | uniq -c | sort -k2,2n

      2 dm-uuid-mpath-360050768108101eb000000000000080f
      2 dm-uuid-mpath-360050768108101eb0000000000000810
      2 dm-uuid-mpath-360050768108101eb0000000000000811
      2 dm-uuid-mpath-360050768108101eb0000000000000812
      2 dm-uuid-mpath-360050768108101eb0000000000000813
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb0000023f
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb00000274
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb0000029b
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb000002c9
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb000002ca
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb000002cb
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb0000032a
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb0000034f
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb00000367
      2 dm-uuid-mpath-360060e8012bdcb005040bdcb00000368
      3 dm-uuid-mpath-360060e8012bdcb005040bdcb00000d19

      Volume with same dm on compute-0381:

      volid: 7f1c45a1 / volid: de7b400f

      mpath: 360060e8012bdcb005040bdcb0000032a

      different backends for those 2 volumes:
      MariaDB [cinder]> select host from volumes where id like '7f1%'\G

                                                          • 1. row ***************************
                                                            host: hostgroup@VSPG900_SSD#VSPG900_SSD
                                                            1 row in set (0.000 sec)

      MariaDB [cinder]> select host from volumes where id like 'de7b%'\G

                                                          • 1. row ***************************
                                                            host: hostgroup@VSPG9000#VSPG9000
                                                            1 row in set (0.000 sec)

      mpath: 360060e8012bdcb005040bdcb0000034f

      MariaDB [cinder]> select host from volumes where id like '4894%'\G

                                                          • 1. row ***************************
                                                            host: hostgroup@VSPG9000#VSPG9000
                                                            1 row in set (0.000 sec)

      MariaDB [cinder]> select host from volumes where id like '2af3%'\G

                                                          • 1. row ***************************
                                                            host: hostgroup@VSPG9000#VSPG9000

      MariaDB [cinder]> select * from volume_attachment where volume_id like '2af3%' and deleted=0\G

                                                          • 1. row ***************************
                                                            created_at: 2024-11-26 21:44:56
                                                            updated_at: 2024-11-26 21:45:08
                                                            deleted_at: NULL
                                                            deleted: 0
                                                            id: a2d5e93b-9b2a-4b9a-95f5-fff7ab132adc
                                                            volume_id: 2af36815-2737-4655-9c46-ebf4a70e2789
                                                            attached_host: compute489
                                                            instance_uuid: f1e4a49f-423d-44a5-af36-68d4cb71f81c
                                                            mountpoint: /dev/vde
                                                            attach_time: 2024-11-26 21:44:58
                                                            detach_time: NULL
                                                            attach_mode: null
                                                            attach_status: attached
                                                            connection_info: {"target_wwn": ["50060e8012bdcb51", "50060e8012bdcb40", "50060e8012bdcb41", "50060e8012bdcb50"], "target_discovered": false, "access_mode": "rw", "target_lun": 5, "qos_specs": null, "encrypted": false, "driver_volume_type": "fibre_channel", "attachment_id": "a2d5e93b-9b2a-4b9a-95f5-fff7ab132adc"}

                                                            connector:

                                                            {"platform": "x86_64", "os_type": "linux", "ip": "10.8.8.39", "host": "compute489", "multipath": true, "initiator": "iqn.1994-05.com.redhat:d961bdbc1bf4", "wwpns": ["100000620b3b311e", "100000620b3b311f"], "wwnns": ["200000620b3b311e", "200000620b3b311f"], "do_local_attach": false, "system uuid": "21f54d5a-c921-11ed-af3f-72e28429c75b", "mountpoint": "/dev/vde", "mode": null}

                                                            1 row in set (0.000 sec)

      MariaDB [cinder]> select * from volume_attachment where volume_id like '4894%' and deleted=0\G

                                                          • 1. row ***************************
                                                            created_at: 2024-11-26 15:39:56
                                                            updated_at: 2024-11-26 15:40:00
                                                            deleted_at: NULL
                                                            deleted: 0
                                                            id: d144d291-1a4a-4776-b8d8-e98bf6a20ac1
                                                            volume_id: 48942692-1fcc-4cbe-b707-49f29a20bcf5
                                                            attached_host: compute0489
                                                            instance_uuid: e4fe6bcb-44eb-4125-98eb-a5cad0c05708
                                                            mountpoint: /dev/vdc
                                                            attach_time: 2024-11-26 15:39:58
                                                            detach_time: NULL
                                                            attach_mode: rw
                                                            attach_status: attached
                                                            connection_info: {"target_wwn": ["50060e8012bdcb51", "50060e8012bdcb40", "50060e8012bdcb41", "50060e8012bdcb50"], "target_discovered": false, "access_mode": "rw", "target_lun": 5, "qos_specs": null, "encrypted": false, "driver_volume_type": "fibre_channel", "attachment_id": "d144d291-1a4a-4776-b8d8-e98bf6a20ac1"}

                                                            connector:

                                                            {"platform": "x86_64", "os_type": "linux", "ip": "10.8.8.39", "host": "compute489", "multipath": true, "initiator": "iqn.1994-05.com.redhat:d961bdbc1bf4", "wwpns": ["100000620b3b311e", "100000620b3b311f"], "wwnns": ["200000620b3b311e", "200000620b3b311f"], "do_local_attach": false, "system uuid": "21f54d5a-c921-11ed-af3f-72e28429c75b", "mountpoint": "/dev/vdc"}

              brosmait@redhat.com Brian Rosmaita
              jira-bugzilla-migration RH Bugzilla Integration
              Yosi Ben Shimon Yosi Ben Shimon
              rhos-storage-cinder
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: