-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
rhos-17.1.3
-
None
-
False
-
-
False
-
?
-
None
-
-
-
Moderate
To Reproduce Steps to reproduce the behavior:
- Cinder backend is external FC storage with multipath
- Glance Backend is cinder
- Nova ephemeral volume backend is the default local storage with qcow2 format
- It fails to create instance snapshots from image boot instances
Issue Description
- When we create an instance snapshot from an image boot instance, nova-compute calls Glance "PUT /v2/images/{image_id}/file" API to upload the image data.
- Glance shows "Since image size is zero, we will be doing resize-before-write for each GB which will be considerably slower than normal" and creates an cinder volume with 1GB initially, and then tries to increase the size of the volume by 1GB repeatedly until it reaches the required size. Additionally, it detach and re-attach the volume every time. Then It looks ugly to me.
And it finally fails with "Timeout while waiting while volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to resize the device in 6 tries" error
Glance log excerpt 13:32:51.482 ... Creating a new volume: image_size=0 size_gb=1 type=None add /usr/lib/python3.9/site-packages/glance_store/_drivers/cinder.py:883 13:32:51.482 ... Since image size is zero, we will be doing resize-before-write for each GB which will be considerably slower than normal. 13:32:55.964 ... ==> connect_volume: 13:32:58.062 ... <== connect_volume: 13:33:04.293 ... ==> disconnect_volume: 13:33:04.631 ... <== disconnect_volume: 13:33:05.515 ... Extending volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to 2 GB. add /usr/lib/python3.9/site-packages/glance_store/_drivers/cinder.py:935 13:33:08.341 ... ==> connect_volume: 13:33:10.434 ... <== connect_volume: 13:33:17.216 ... ==> disconnect_volume: 13:33:17.594 ... <== disconnect_volume: 13:33:18.535 ... Extending volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to 3 GB. add /usr/lib/python3.9/site-packages/glance_store/_drivers/cinder.py:935 13:33:20.979 ... ==> connect_volume: 13:33:23.062 ... <== connect_volume: 13:33:30.073 ... ==> disconnect_volume: 13:33:30.322 ... <== disconnect_volume: 13:33:31.304 ... Extending volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to 4 GB. add /usr/lib/python3.9/site-packages/glance_store/_drivers/cinder.py:935 13:33:33.826 ... ==> connect_volume: 13:33:39.949 ... <== connect_volume: 13:34:05.705 ... Timeout while waiting while volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to resize the device in 6 tries. 13:34:05.947 ... Exception while accessing to cinder volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa.: glance_store.exceptions.BackendException: Timeout while waiting while volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to resize the device in 6 tries. 13:34:05.952 ... ==> disconnect_volume: 13:34:18.239 ... <== disconnect_volume: 13:34:19.129 ... Failed to write to volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa.: glance_store.exceptions.BackendException: Timeout while waiting while volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to resize the device in 6 tries. 13:34:19.282 ... Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Timeout while waiting while volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to resize the device in 6 tries. 13:34:19.352 ... Caught error: Timeout while waiting while volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to resize the device in 6 tries.: glance_store.exceptions.BackendException: Timeout while waiting while volume aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaa to resize the device in 6 tries.
Expected behavior
- Glance creates the cinder volume with its proper size from the beginning, and instance snapshot creation works
Bug impact
- Cannot create instance snapshots from image boot instances
Known workaround
- None
Requested action:
- Are there any workarounds?
- Why does Glance create a cinder volume with 1GB initially, and then try to increase the size of the volume by 1GB repeatedly until it reaches the required size?
I don't think it's smart...
