Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-20530

Different workers from single Glance API service may issue conflicting calls when working with same volume-based images

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • rhos-17.1.z, rhos-17.1.9
    • openstack-glance
    • None
    • 13
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • None
    • Critical

      To Reproduce Steps to reproduce the behavior:
      We have found another problem similar to https://issues.redhat.com/browse/OSPRH-19973 , but this time it is caused by conflicting calls issues by different workers from the same Glance API service. Please find more details below.

      Logs will be provided privately and problem may manifest itself in different ways as well. What we saw is:

      • one Glance worker attached volume to process a call against volume-backed image
      • another Glance worker tried to attach volume to process another call slightly later
      • first worker completed its operation and disconnected volume
      • second worker failed its operation with the following error:
        2025-10-02 16:48:44.038 44 ERROR glance_store._drivers.cinder [req-b6194db5-86d5-4994-bafc-277b8b61b80b ] Exception while accessing to cinder volume 525eb3eb-7375-4f37-914a-07103ec3be74.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
        Command: sudo glance-rootwrap /etc/glance/rootwrap.conf chown 0 /dev/disk/by-id/emc-vol-10fdd887757db40f-0bfcde3900000120
        Exit code: 1
        Stdout: ''
        

      Expected behavior
      A Glance API locking mechanism prevents this situation

      Bug impact
      Some VMs will not be created when batch of VMs are started using same cinder-based Glance image, but this may affect other operations as well.

      Known workaround
      We have solved this by decreasing amount of Glance APi workers to 1 and by separating glance_api and glance_api_internal services to address https://issues.redhat.com/browse/OSPRH-19973 as well. But this is not suitable for production.

      Additional context
      To be provided privately

              rdhasman@redhat.com Rajat Dhasmana
              rhn-support-astupnik Alex Stupnikov
              rhos-storage-glance
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: