-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
2025.2 (Flamingo), rhos-18.0.10 FR 3
-
None
Bug Description
Glance over Cinder/NFS configuration randomly fails with the following message when create/delete image operations are performed concurrently. This happens because the current lock mechanism in glance_store that prevents race conditions on volume mount/unmount [1][2] is local to the process, and fails when multiple Glance workers run concurrently.
Jun 03 19:37:41.316051 np0041005539 devstack@g-api.service[90321]: WARNING glance.image_cache.drivers.centralized_db [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': FileExistsError(17, 'File exists'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/bc2f9475-0d11-4f96-b8e6-b22fdce5cace', 'invalid_path': '/opt/stack/data/glance/cache/invalid/bc2f9475-0d11-4f96-b8e6-b22fdce5cace'}): FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache [None req-3aff3146-9e2a-4de4-9f7b-4d8d622c03da tempest-ImageDependencyTests-1910387855 tempest-ImageDependencyTests-1910387855-project-member] Exception encountered while tee'ing image 'bc2f9475-0d11-4f96-b8e6-b22fdce5cace' into cache: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'. Continuing with response.: FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache Traceback (most recent call last):
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/opt/stack/glance/glance/image_cache/__init__.py", line 360, in cache_tee_iter
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache for chunk in image_iter:
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache for chunk in data:
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 813, in _cinder_volume_data_iterator
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache with self._open_cinder_volume(client, volume, 'rb') as fp:
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache return next(self.gen)
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/store.py", line 770, in _open_cinder_volume
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache device = conn.connect_volume(volume)
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 03 19:37:41.348228 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 94, in connect_volume
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache device = connect_volume_nfs()
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^^^^^^^
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py", line 415, in inner
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache return f(*args, **kwargs)
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache ^^^^^^^^^^^^^^^^^^
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/_drivers/cinder/nfs.py", line 89, in connect_volume_nfs
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache mount.mount(
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 358, in mount
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache mount_state.mount(fstype, export, vol_name, mountpoint, host,
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "/opt/stack/data/venv/lib/python3.12/site-packages/glance_store/common/fs_mount.py", line 247, in mount
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache os.makedirs(mountpoint)
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache File "<frozen os>", line 225, in makedirs
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache FileExistsError: [Errno 17] File exists: '/opt/stack/data/glance/images/mnt/8db6edfa5db8c4a4b76e2ad4526a0f97806edb8c809ce63ffcdb8d1a551aa57a'
Jun 03 19:37:41.348840 np0041005539 devstack@g-api.service[90321]: ERROR glance.image_cache
Steps to reproduce the behavior:
- Try to run multiple create/delete image operations concurrently in an environment configured with Glance over Cinder/NFS.
Expected behavior
- All image create/delete operations should complete without errors.
Bug impact
- CI jobs are failing;
- Image create behavior is unreliable for environments running Glance over Cinder/NFS
Known workaround
- For the Glance service, set the number of workers to 1. This has a significant performance impact, but currently it is the only way to mitigate the issue.
