Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-9726

[NetApp NFS] Glance with Cinder NFS backend fails while verifying image's virtual size vs cinder volume size

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • ?
    • ?
    • None
    • Important

      Glance with Cinder NFS backend fails while verifying image's virtual size vs cinder volume size

      Steps:
      1. Have NFS based backends in Cinder
      2. Configure Glance to use Cinder NFS based backend (store=cinder)
      3. Upload the image to Glance Cinder Store
      4. Check image is in active state
      5. Now, try to do one of the following opearations:

      a. Create a boot volume from the image
      b. Create a server from the image (as source)
      c. Try to save the image to local location

      In all the above workflows, the operation would fail with the below error message:

      ERROR cinder.volume.drivers.nfs [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit. Virtual Size is 117440512 and real size is 1

      There is a below check which has been added recently and IMO it causes the problem.

      https://github.com/openstack/cinder/commit/78f85c1f9b20a067ef64d6451dee0228c3a0db5e#diff-3756fbc353639b4ae5ed59cf47fb9ed72e3330b21d5165cd797d73ee21554481

      When the image is uploaded to cinder store, by default cinder creates 1GB volume. That case, the virtual size of the image will not exactly match with the cinder volume size. This will result into the InvalidVolume exception.

      Cinder volume spec when the image is uploaded:
      ==================================================
      DEBUG cinder.volume.manager [[[01;36mNone req-316fef3e-fc35-4dd6-914d-2955373914c0 ^[[00;36mservice None] ^[[01;35mTask 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (6c4edd58-7186-418a-aa27-202a53a7e3a0) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2024-07-15T18:16:20Z,deleted=False,deleted_at=None,display_description=None,display_name='image-4a6fbdf5-afa4-4c14-be4c-e2d625b37e33',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host='ostack-spare3.nb.openenglab.netapp.com@nbnfs#10.195.15.47:/glance_nfs_vol',id=f53989d0-5008-4578-8299-5c0b9b6839bf,launched_at=None,metadata={glance_image_id='4a6fbdf5-afa4-4c14-be4c-e2d625b37e33',image_owner='681b668eb8564bc8b3a0367edd179461',image_size='21233664'},migration_status=None,multiattach=False,previous_status=None,project_id='8b0b3a24c3494456b62900e4f13a4d21',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=2024-07-15T18:16:20Z,service_uuid=None,shared_targets=True,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2024-07-15T18:16:20Z,use_quota=True,user_id='f2ecefb8558c4c1dbbd043fcf61969cb',volume_attachment=VolumeAttachmentList,volume_type=VolumeType(714c876f-99af-46a7-9f65-955478a5b1a8),volume_type_id=714c876f-99af-46a7-9f65-955478a5b1a8)'[[00m ^[[00;33m{{(pid=2777261) _task_receiver /opt/stack/data/venv/lib/python3.10/site-packages/taskflow/listeners/logging.py:178}}

      stack@ostack-spare3:~$ qemu-img info cirros-0.6.2-x86_64-disk.img
      image: cirros-0.6.2-x86_64-disk.img
      file format: qcow2
      virtual size: 112 MiB (117440512 bytes)
      disk size: 20.4 MiB
      cluster_size: 65536
      Format specific information:
          compat: 1.1
          compression type: zlib
          lazy refcounts: false
          refcount bits: 16
          corrupt: false
          extended l2: false
      stack@ostack-spare3:~$

      stack@ostack-spare3:~$ cinder list --all | grep 4a6fbdf5-afa4-4c14-be4c-e2d625b37e33

      f53989d0-5008-4578-8299-5c0b9b6839bf 8b0b3a24c3494456b62900e4f13a4d21 available image-4a6fbdf5-afa4-4c14-be4c-e2d625b37e33 1 msnfs false  

      stack@ostack-spare3:~$

      Actual Logs from cinder-volume.log:
      ===================================
      DEBUG oslo_concurrency.processutils [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] CMD "/opt/stack/data/venv/bin/python3.10 -m oslo_concurrency.prlimit --as=1073741824 --cpu=60 – sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info -f qcow2 --output=json --force-share /opt/stack/data/cinder/mnt/d1cb57de567907812dd76ef18aafeba4/volume-82284f9a-f10e-47c2-bc3a-0818bb741b44" returned: 0 in 0.392s (pid=2767016) execute /opt/stack/data/venv/lib/python3.10/site-packages/oslo_concurrency/processutils.py:428
      ERROR cinder.volume.drivers.nfs [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit. Virtual Size is 117440512 and real size is 1
      ERROR cinder.volume.manager [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] Driver initialize connection failed (error: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.).: cinder.exception.InvalidVolume: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.
      ERROR cinder.volume.manager Traceback (most recent call last):
      ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/manager.py", line 4832, in _connection_create
      ERROR cinder.volume.manager conn_info = self.driver.initialize_connection(volume, connector)
      ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 170, in initialize_connection
      ERROR cinder.volume.manager raise exception.InvalidVolume(reason=msg)
      ERROR cinder.volume.manager cinder.exception.InvalidVolume: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.
      ERROR cinder.volume.manager
      ERROR oslo_messaging.rpc.server [req-dc5f0258-7402-4236-8251-38eb45e1a305 req-6088ee76-f616-4ec2-b696-d17d9acfd609 service None] Exception during message handling: cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.).
      ERROR oslo_messaging.rpc.server Traceback (most recent call last):
      ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4832, in _connection_create
      ERROR oslo_messaging.rpc.server conn_info = self.driver.initialize_connection(volume, connector)
      ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/drivers/nfs.py", line 170, in initialize_connection
      ERROR oslo_messaging.rpc.server raise exception.InvalidVolume(reason=msg)
      ERROR oslo_messaging.rpc.server cinder.exception.InvalidVolume: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.
      ERROR oslo_messaging.rpc.server
      ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
      ERROR oslo_messaging.rpc.server
      ERROR oslo_messaging.rpc.server Traceback (most recent call last):
      ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
      ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
      ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
      ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
      ERROR oslo_messaging.rpc.server File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
      ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
      ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4889, in attachment_update
      ERROR oslo_messaging.rpc.server connection_info = self._connection_create(context,
      ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/volume/manager.py", line 4838, in _connection_create
      ERROR oslo_messaging.rpc.server raise exception.VolumeBackendAPIException(data=err_msg)
      ERROR oslo_messaging.rpc.server cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Invalid volume: The volume virtual_size does not match the size in cinder, aborting as we suspect an exploit.).
      ERROR oslo_messaging.rpc.server

            brosmait@redhat.com Brian Rosmaita
            udesale@redhat.com Unmesh Desale
            rhos-dfg-storage-squad-cinder
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: