Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: rhos-18.0.0
Affects Version/s: None
Component/s: cinder-operator
Labels:
- rhos-18-beta-waive

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Dev Approval:
?
Docs Approval:
?
Fixed in Build:
cinder-operator-bundle-container-1.0.0-44
PM Approval:
?
QE Approval:
?
Regression:
None
Release Note Text:

Hide
.Image uploads might fail if a multipathing path for Block Storage service (cinder) volumes is offline

If you use multipath for Block storage service volumes, and you have configured the Block Storage service as the back end for the Image service (glance), image uploads might fail if one of the paths goes offline.

Show
.Image uploads might fail if a multipathing path for Block Storage service (cinder) volumes is offline If you use multipath for Block storage service volumes, and you have configured the Block Storage service as the back end for the Image service (glance), image uploads might fail if one of the paths goes offline.
Release Note Type:
Release Note Not Required
Release Note Status:
Done
Status Summary:

Hide

RHOSO18Beta waived:Storage: Cinder multipath

Show
RHOSO18Beta waived: Storage: Cinder multipath
Test Coverage:

Automated
Intelligence Requested:
Market:
Release Blocker:
Approved

Severity:
Critical

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

We observe the failures when Glance is configured with Cinder backend and configured to use multipath. The problem appears when trying to upload the image.
-> https://github.com/openstack-k8s-operators/architecture/blob/main/examples/dt/uni05epsilon/control-plane/service-values.yaml#L68

The multipath was configured on all ocp nodes with devscripts.

This is to be investigated.

2024-05-30 08:42:05.009 3085181 DEBUG os_brick.utils [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] <== get_connector_properties: return (3230ms) {'platform': 'x86_64', 'os_type': 'linux', 'ip': '192.168.17.84', 'host': 'glance-default-external-api-0', 'multipath': True, 'initiator': 'iqn.1994-05.com.redhat:bb21eadff3b3', 'do_local_attach': False, 'nvme_hostid': '22bdc55d-1181-41b5-af54-da5d9e7ec25e', 'system uuid': '7f2f9c98-9479-4537-8fc9-590531bb0b91', 'nqn': 'nqn.2014-08.org.nvmexpress:uuid:a9e88c4d-a632-4acf-b0bb-9f817e4c5195'[0/1066]
native_multipath': True, 'found_dsc': ''} trace_logging_wrapper /usr/lib/python3.9/site-packages/os_brick/utils.py:203
2024-05-30 08:42:05.184 3085181 DEBUG glance_store._drivers.cinder.store [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] Attachment 45c5fe3d-7cf3-4cc5-a8c5-07e18747021e created successfully. _open_cinder_volume /usr/lib/python3.9/site-packages/glance_store/_drivers/cinder/store.py:723
2024-05-30 08:42:05.686 3085181 ERROR glance_store.common.cinder_utils [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] Update attachment failed for attachment 45c5fe3d-7cf3-4cc5-a8c5-07e18747021e. Error: Unable to update the attachment. (HTTP 500) (Request-ID: req-fd5cdaa1-5c87-4ca3-b03f-658ba40ccd76) Code: 500: cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-fd5cdaa1-5c87-4ca3-b03f-658ba40ccd76)
2024-05-30 08:42:05.687 3085181 ERROR glance_store._drivers.cinder.store [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] Failed to write to volume 5d262baa-7d62-4b7b-9871-1cbb5b08570c.: cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-fd5cdaa1-5c87-4ca3-b03f-658ba40ccd76)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] Failed to delete of volume 5d262baa-7d62-4b7b-9871-1cbb5b08570c.: cinderclient.exceptions.BadRequest: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots, awaiting a transfer, or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-50d39bb2-2761-4fef-beac-0c21726
162b5)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store Traceback (most recent call last):
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 1037, in add
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     self._offline_extend(client, volume, write_props)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 931, in _offline_extend
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     with self._open_cinder_volume(client, volume, 'wb') as f:
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     return next(self.gen)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 725, in _open_cinder_volume
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     attachment = self.volume_api.attachment_update(
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/glance_store/common/cinder_utils.py", line 35, in wrapper
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     res = method(self, ctx, volume_id, *args, **kwargs)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/glance_store/common/cinder_utils.py", line 171, in attachment_update
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     LOG.error(_LE('Update attachment failed for attachment '
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     self.force_reraise()
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     raise self.value
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/glance_store/common/cinder_utils.py", line 166, in attachment_update
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     attachment_ref = client.attachments.update(
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/api_versions.py", line 421, in substitution
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     return method.func(obj, *args, **kwargs)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/v3/attachments.py", line 75, in update
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     resp = self._update('/attachments/%s' % id, body)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/base.py", line 318, in _update
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     resp, body = self.api.client.put(url, body=body, **kwargs)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 226, in put
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     return self._cs_request(url, 'PUT', **kwargs)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 211, in _cs_request
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     return self.request(url, method, **kwargs)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 197, in request
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     raise exceptions.from_response(resp, body)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-fd5cdaa1-5c87-4ca3-b03f-658ba40ccd76)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store During handling of the above exception, another exception occurred:
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store Traceback (most recent call last):
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 1050, in add
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     volume.delete()
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/v3/volumes_base.py", line 29, in delete
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     return self.manager.delete(self, cascade=cascade)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/v3/volumes_base.py", line 235, in delete
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     return self._delete(loc)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/base.py", line 313, in _delete
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     resp, body = self.api.client.delete(url)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 229, in delete
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     return self._cs_request(url, 'DELETE', **kwargs)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 211, in _cs_request
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     return self.request(url, method, **kwargs)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 197, in request
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store     raise exceptions.from_response(resp, body)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store cinderclient.exceptions.BadRequest: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots, awaiting a transfer, or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-50d39bb2-2761-4fef-beac-0c21726162b5)
2024-05-30 08:42:05.836 3085181 ERROR glance_store._drivers.cinder.store
2024-05-30 08:42:05.843 3085181 ERROR glance.api.v2.image_data [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] Failed to upload image data due to internal error: cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-fd5cdaa1-5c87-4ca3-b03f-658ba40ccd76)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] Caught error: Unable to update the attachment. (HTTP 500) (Request-ID: req-fd5cdaa1-5c87-4ca3-b03f-658ba40ccd76): cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-fd5cdaa1-5c87-4ca3-b03f-658ba40ccd76)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi Traceback (most recent call last):
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/common/wsgi.py", line 1286, in __call__
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     action_result = self.dispatch(self.controller, action,
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/common/wsgi.py", line 1329, in dispatch
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     return method(*args, **kwargs)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/common/utils.py", line 414, in wrapped
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/api/v2/image_data.py", line 302, in upload
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     self._restore(image_repo, image)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     self.force_reraise()
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     raise self.value
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/api/v2/image_data.py", line 162, in upload
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     image.set_data(data, size, backend=backend)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/notifier.py", line 497, in set_data
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     self.force_reraise()
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     raise self.value
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/notifier.py", line 443, in set_data
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     self.repo.set_data(data, size, backend=backend,
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/quota/__init__.py", line 322, in set_data
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     self.image.set_data(data, size=size, backend=backend,
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/location.py", line 585, in set_data
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     self._upload_to_store(data, verifier, backend, size)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance/location.py", line 476, in _upload_to_store
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     multihash, loc_meta) = self.store_api.add_with_multihash(
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/multi_backend.py", line 397, in add_with_multihash
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     return store_add_to_backend_with_multihash(
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/multi_backend.py", line 479, in store_add_to_backend_with_multihash
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     (location, size, checksum, multihash, metadata) = store.add(
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/driver.py", line 277, in add_adapter
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     metadata_dict) = store_add_fun(*args, **kwargs)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/capabilities.py", line 176, in op_checker
2024-05-30 08:42:05.872 30851https://github.com/openstack-k8s-operators/architecture/blob/main/examples/dt/uni05epsilon/control-plane/service-values.yaml#L6881 ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 1037, in add
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     self._offline_extend(client, volume, write_props)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 931, in _offline_extend
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     with self._open_cinder_volume(client, volume, 'wb') as f:
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib64/python3.9/contextlib.py", line 119, in __enter__
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     return next(self.gen)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/_drivers/cinder/store.py", line 725, in _open_cinder_volume
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     attachment = self.volume_api.attachment_update(
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/common/cinder_utils.py", line 35, in wrapper
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     res = method(self, ctx, volume_id, *args, **kwargs)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/common/cinder_utils.py", line 171, in attachment_update
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     LOG.error(_LE('Update attachment failed for attachment '
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     self.force_reraise()
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     raise self.value
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/glance_store/common/cinder_utils.py", line 166, in attachment_update
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     attachment_ref = client.attachments.update(
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/cinderclient/api_versions.py", line 421, in substitution
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     return method.func(obj, *args, **kwargs)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/cinderclient/v3/attachments.py", line 75, in update
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     resp = self._update('/attachments/%s' % id, body)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/cinderclient/base.py", line 318, in _update
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     resp, body = self.api.client.put(url, body=body, **kwargs)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 226, in put
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     return self._cs_request(url, 'PUT', **kwargs)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 211, in _cs_request
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     return self.request(url, method, **kwargs)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 197, in request
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi     raise exceptions.from_response(resp, body)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-fd5cdaa1-5c87-4ca3-b03f-658ba40ccd76)
2024-05-30 08:42:05.872 3085181 ERROR glance.common.wsgi
2024-05-30 08:42:05.936 3085181 INFO eventlet.wsgi.server [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] 192.168.111.21,192.168.16.2,::1 - - [30/May/2024 08:42:05] "PUT /v2/images/ff5de724-f0ee-4e63-859e-8d353823e1ba/file HTTP/1.1" 500 341 6.271686
2024-05-30 08:42:05.948 3085181 DEBUG glance.api.middleware.version_negotiation [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] Determining version of request: DELETE /v2/images/ff5de724-f0ee-4e63-859e-8d353823e1ba Accept: */* process_request /usr/lib/python3.9/site-packages/glance/api/middleware/version_negotiation.py:44
2024-05-30 08:42:05.948 3085181 DEBUG glance.api.middleware.version_negotiation [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] Using url versioning process_request /usr/lib/python3.9/site-packages/glance/api/middleware/version_negotiation.py:57
2024-05-30 08:42:05.949 3085181 DEBUG glance.api.middleware.version_negotiation [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] Matched version: v2 process_request /usr/lib/python3.9/site-packages/glance/api/middleware/version_negotiation.py:69
2024-05-30 08:42:05.949 3085181 DEBUG glance.api.middleware.version_negotiation [None req-24d48df9-8a42-43f5-8db5-58cedebbf615 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] new path /v2/images/ff5de724-f0ee-4e63-859e-8d353823e1ba process_request /usr/lib/python3.9/site-packages/glance/api/middleware/version_negotiation.py:70
2024-05-30 08:42:06.003 3085181 INFO eventlet.wsgi.server [None req-a60b5032-3d1f-4c1f-8f84-3606c79df0f2 5d7966977e5e40e399f2b6bbe1e9a7bc e34b8c9db6d44da6a618d8073dd3ce31 - - default default] 192.168.111.21,192.168.16.2,::1 - - [30/May/2024 08:42:06] "DELETE /v2/images/ff5de724-f0ee-4e63-859e-8d353823e1ba HTTP/1.1" 204 213 0.055336
 
[0] 0:[tmux]*Z 1:ssh- 2:ssh                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      "titan61.lab.eng.tlv2." 1