Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-12844

BZ#2230602 [RHOS-17.1] Resizing compute with NVMeOF Cinder backend intermittently fails to find volume

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • rhos-17.1.0, rhos-18.0.0
    • python-os-brick
    • False
    • Hide

      None

      Show
      None
    • False
    • None

      Description of problem: Resizing a guest with NVMeOF Cinder backend in intermittently failing in phase3 CI. Test specifically triggering this failure [1]. Full logs, deployment details, and job details will be included in follow up comment. Please advise if the attempted actions are unsupported for the environment.

      2023-08-03 02:01:55.660 18 DEBUG placement.objects.research_context [req-37c55e62-a4c1-41d6-be37-e83b9dbe7364 d296552183034fba8b15267e5b19b68d ddccfaba37ea41a4bc85a4a418f8f9e2 - default default] found 2 providers after filtering by previous result get_provider_ids_matching /usr/lib/python3.9/site-packages/placement/objects/research_context.py:580
      2023-08-03 02:01:55.661 2 DEBUG os_brick.initiator.connectors.base [req-a9e03d62-5284-4fbf-bdc8-c2bcc78d4703 571ccebe0b70413691998e6cac3046ca e40a3908acf04d27b9e8f4bbbe5005ff - default default] Lock "connect_volume" "released" by "os_brick.initiator.connectors.nvmeof.NVMeOFConnector.connect_volume" :: held 6.036s inner /usr/lib/python3.9/site-packages/os_brick/initiator/connectors/base.py:83
      2023-08-03 02:01:55.661 2 DEBUG os_brick.initiator.connectors.nvmeof [req-a9e03d62-5284-4fbf-bdc8-c2bcc78d4703 571ccebe0b70413691998e6cac3046ca e40a3908acf04d27b9e8f4bbbe5005ff - default default] <== connect_volume: exception (6037ms) VolumeDeviceNotFound('Volume device not found at nqn.nvme-subsystem-compute-0.redhat.local.') trace_logging_wrapper /usr/lib/python3.9/site-packages/os_brick/utils.py:176
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [req-a9e03d62-5284-4fbf-bdc8-c2bcc78d4703 571ccebe0b70413691998e6cac3046ca e40a3908acf04d27b9e8f4bbbe5005ff - default default] [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] Setting instance vm_state to ERROR: os_brick.exception.VolumeDeviceNotFound: Volume device not found at nqn.nvme-subsystem-compute-0.redhat.local.
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] Traceback (most recent call last):
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 10256, in _error_out_instance_on_exception
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] yield
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5851, in _finish_resize_helper
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] network_info = self._finish_resize(context, instance, migration,

      ....<REMOVED FOR BREVITY>....

      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] result = fn(*args, **kwargs)
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] File "/usr/lib/python3.9/site-packages/os_brick/initiator/connectors/nvmeof.py", line 908, in _connect_target
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] raise exception.VolumeDeviceNotFound(device=target.nqn)
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea] os_brick.exception.VolumeDeviceNotFound: Volume device not found at nqn.nvme-subsystem-compute-0.redhat.local.
      2023-08-03 02:01:55.662 2 ERROR nova.compute.manager [instance: 2f89eb85-eccc-4b9c-a500-490237c40fea]

      Version-Release number of selected component (if applicable):
      RHOS-17.1-RHEL-9-20230712.n.1

      How reproducible:
      50%

      Steps to Reproduce:
      1. Deploy 17.1 with NVMeOF Cinder backend enabled
      2. Create a server and volume and attach the volume to the server
      3. Resize the server

      Actual results:
      Server resize is failing due to attach volume not being found during resize process

      Expected results:
      server resize action is successful

      Additional info:
      [1] https://github.com/openstack/tempest/blob/34.1.0/tempest/api/compute/servers/test_server_actions.py#L459C1-L459C58

              Unassigned Unassigned
              jira-bugzilla-migration RH Bugzilla Integration
              Yosi Ben Shimon Yosi Ben Shimon
              rhos-dfg-storage-squad-cinder
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: