-
Bug
-
Resolution: Unresolved
-
Critical
-
rhos-18.0.0
-
False
-
-
False
-
?
-
?
-
?
-
?
-
None
-
-
-
Important
Glance with Cinder NFS backend fails while verifying image's virtual size vs cinder volume size
Steps:
1. Have NFS based backends in Cinder
2. Configure Glance to use Cinder NFS based backend (store=cinder)
3. Upload the image to Glance Cinder Store
4. Check image is in active state
5. Now, try to do one of the following opearations:
a. Create a boot volume from the image
b. Create a server from the image (as source)
c. Try to save the image to local location
In all the above workflows, the operation would fail with the below error message:
ERROR cinder.volume.drivers.nfs [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit. Virtual Size is 117440512 and real size is 1
There is a below check which has been added recently and IMO it causes the problem.
When the image is uploaded to cinder store, by default cinder creates 1GB volume. That case, the virtual size of the image will not exactly match with the cinder volume size. This will result into the InvalidVolume exception.
Cinder volume spec when the image is uploaded:
==================================================
DEBUG cinder.volume.manager [[[01;36mNone req-316fef3e-fc35-4dd6-914d-2955373914c0 ^[[00;36mservice None] ^[[01;35mTask 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6c4edd58-7186-418a-aa27-202a53a7e3a0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2024-07-15T18:16:20Z,deleted=False,deleted_at=None,display_description=None,display_name='image-4a6fbdf5-afa4-4c14-be4c-e2d625b37e33',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ostack-spare3.nb.openenglab.netapp.com@nbnfs#10.195.15.47:/glance_nfs_vol',id=f53989d0-5008-4578-8299-5c0b9b6839bf,launched_at=None,metadata={glance_image_id='4a6fbdf5-afa4-4c14-be4c-e2d625b37e33',image_owner='681b668eb8564bc8b3a0367edd179461',image_size='21233664'},migration_status=None,multiattach=False,previous_status=None,project_id='8b0b3a24c3494456b62900e4f13a4d21',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2024-07-15T18:16:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2024-07-15T18:16:20Z,use_quota=True,user_id='f2ecefb8558c4c1dbbd043fcf61969cb',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(714c876f-99af-46a7-9f65-955478a5b1a8),volume_type_id=714c876f-99af-46a7-9f65-955478a5b1a8)'[[00m ^[[00;33m{{(pid=2777261) _task_receiver /opt/stack/data/venv/lib/python3.10/site-packages/taskflow/listeners/logging.py:178}}
stack@ostack-spare3:~$ qemu-img info cirros-0.6.2-x86_64-disk.img
image: cirros-0.6.2-x86_64-disk.img
file format: qcow2
virtual size: 112 MiB (117440512 bytes)
disk size: 20.4 MiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
extended l2: false
stack@ostack-spare3:~$
stack@ostack-spare3:~$ cinder list --all | grep 4a6fbdf5-afa4-4c14-be4c-e2d625b37e33
f53989d0-5008-4578-8299-5c0b9b6839bf | 8b0b3a24c3494456b62900e4f13a4d21 | available | image-4a6fbdf5-afa4-4c14-be4c-e2d625b37e33 | 1 | msnfs | false |
stack@ostack-spare3:~$
Actual Logs from cinder-volume.log:
===================================
DEBUG oslo_concurrency.processutils [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] CMD "/opt/stack/data/venv/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 – sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/d1cb57de567907812dd76ef18aafeba4/volume-82284f9a-f10e-47c2-bc3a-0818bb741b44" returned: 0 in 0.392s (pid=2767016) execute /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/processutils.py:428
ERROR cinder.volume.drivers.nfs [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit. Virtual Size is 117440512 and real size is 1
ERROR cinder.volume.manager [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] Driver initialize connection failed (error: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.).: cinder.exception.InvalidVolume: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.
ERROR cinder.volume.manager Traceback (most recent call last):
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 4832, in _connection_create
ERROR cinder.volume.manager conn_info = self.driver.initialize_connection(volume, connector)
ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 170, in initialize_connection
ERROR cinder.volume.manager raise exception.InvalidVolume(reason=msg)
ERROR cinder.volume.manager cinder.exception.InvalidVolume: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.
ERROR cinder.volume.manager
ERROR oslo_messaging.rpc.server [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] Exception during message handling: cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.).
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4832, in _connection_create
ERROR oslo_messaging.rpc.server conn_info = self.driver.initialize_connection(volume, connector)
ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 170, in initialize_connection
ERROR oslo_messaging.rpc.server raise exception.InvalidVolume(reason=msg)
ERROR oslo_messaging.rpc.server cinder.exception.InvalidVolume: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
ERROR oslo_messaging.rpc.server
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4889, in attachment_update
ERROR oslo_messaging.rpc.server connection_info = self._connection_create(context,
ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4838, in _connection_create
ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=err_msg)
ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.).
ERROR oslo_messaging.rpc.server