Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-13025

BZ#2299636 Some multipath device paths are missing after instance live migration with FC cinder backend

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • rhos-16.2.z
    • python-os-brick
    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • Important

      Description of problem:

      After instance live-migration, cinder volume device attachments are also migrated to the destination compute node.
      However, sometimes a multipath device has fewer paths than expected.

      ~~~
      36xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx dm-x XXXX,XXXXXXX
      size=100G features='0' hwhandler='1 alua' wp=rw
      `+ policy='service-time 0' prio=50 status=active

      • 1:0:3:1 sde 8:64 active ready running
        `- 2:0:3:1 sdg 8:96 active ready running

      ==> This multipath device is corresponding to a cinder volume attached to the migrated instance
      ==> This multipath device should have 4 paths, but actually it only has 2 paths
      ~~~

      After checking logs, I noticed that some of paths were detected automatically, but other paths were not detected automatically.
      When there is no paths in a multipath device, os-brick scans all paths[1] and all paths are detected. In this case, the multipath device has all paths properly.
      However, when only some paths of a multipath device are automatically detected and other paths are not, os-brick doesn't scan other paths, because os-brick doesn't rescan when there are at least one detected path[2] in a multipath device. In this case, some multipath device paths are missing.
      When we manually scan all paths using `echo "- - -" > /sys/class/scsi_host/hostX/scan` command, all paths are detected. So, there are not connectivity issues between the compute node and the backend storage.

      I'm not sure why not all paths were not automatically detected, but I'm wondering if os-brick should scan all paths always even if some paths are already detected.

      Version-Release number of selected component (if applicable):
      The customer reported this issue on RHOSP 16.2.0 + Pure Storage FC backend

      How reproducible:
      Steps to Reproduce:
      1. Create an instance and attach multiple cinder volumes
      2. live-migrate the instance 14 times.
      3. The issue occurred 4 times out of 14 live-migrations

      Actual results:
      Some of multipath device paths are missing sometimes after live-migration

      Expected results:
      All paths of multipath devices are recognized always after live-migration

      Additional info:
      [1] https://github.com/openstack/os-brick/blob/train-eol/os_brick/initiator/connectors/fibre_channel.py#L236
      [2] https://github.com/openstack/os-brick/blob/train-eol/os_brick/initiator/connectors/fibre_channel.py#L218-L226

              rdhasman@redhat.com Rajat Dhasmana
              jira-bugzilla-migration RH Bugzilla Integration
              Yosi Ben Shimon Yosi Ben Shimon
              rhos-storage-cinder
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: