Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-19973

Glance API and Internal Glance API running on the same host with Cinder backend cause race conditions

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • rhos-17.1.z
    • openstack-glance
    • None
    • 13
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • None
    • Critical

      To Reproduce Steps to reproduce the behavior:
      In RHOSP 17.1 we switched to architecture with two Glance services co-located on the same set of controller nodes. It looks like that in some circumstances, when same image is processed by both services simultaneously, a race condition is possible.

      One of our customers faced problems when one of backends reported errors like

      2025-09-05 14:18:05.044 17 ERROR glance_store._drivers.cinder [...] Exception while accessing to cinder volume 453ead5f-a8bf-442e-9343-1c75a82d5e92.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
      Command: sudo glance-rootwrap /etc/glance/rootwrap.conf chown 42415 /dev/disk/by-id/emc-vol-10fdd887757db40f-0bfb055700000039
      Exit code: 1
      Stdout: ''
      Stderr: "chown: cannot access '/dev/disk/by-id/emc-vol-10fdd887757db40f-0bfb055700000039': No such file or directory\n"

      From the logs (will be provided privately) it looks like Internal Glance API failed because Glance API detached the cinder-volume.

      Expected behavior
      I am not sure if we will be able to do something about this. But if not, then IMO this should be documented as known issue.

      Bug impact
      Blocks customer's acceptance tests

      Known workaround
      As a temporary solution we stopped Glance API on 2 controller nodes and stopped Internal Glance API on remaining one. This seem to solve the issue (or at least made it less severe, 2nd look at logs is appreciated), but is not something we could use as a permanent solution for production.

              rdhasman@redhat.com Rajat Dhasmana
              rhn-support-astupnik Alex Stupnikov
              rhos-storage-glance
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: