Loading...

Type: Bug
Resolution: Not a Bug
Priority: Undefined
Fix Version/s: None
Affects Version/s: rhel-9.5
Component/s: virtiofsd
Labels:
None

Regression:
No
Severity:
Important

Pool Team:

rhel-sst-virtualization-storage
Sub-System Group:

ssg_virtualization

Story Points:
None
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Sprint:
None

Preliminary Testing:
None
Test Coverage:
None

Experience:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

What were you trying to do that didn't work?

Migration will failed when there are 2 shared virtiofs directories.

Please provide the package NVR for which bug is seen:

virtiofsd-1.11.1-1.el9

qemu-kvm-9.0.0-7.el9

kernel-5.14.0-480.el9

How reproducible:

100%

Steps to reproduce

1.Create 2 local shared directores in source node

mkdir /home/timao/virtiofsd/local1/

mkdir /home/timao/virtiofsd/local2/

2. Export the 2 directories over virtiofsd

/usr/libexec/virtiofsd --shared-dir /home/timao/virtiofsd/local1/ --socket-path /home/timao/virtiofsd/source_virtiofsd.sock --cache auto &

/usr/libexec/virtiofsd --shared-dir /home/timao/virtiofsd/local2/ --socket-path /home/timao/virtiofsd/source_virtiofsd2.sock --cache auto &

3. Bootup a guest with the 2 shared directory in source node

# /libexec/qemu-kvm \
-S  \
-name 'avocado-vt-vm1'  \
-sandbox on,elevateprivileges=deny,obsolete=deny,resourcecontrol=deny \
-blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
-blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
-blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel950-64-virtio-ovmf_qcow2_filesystem_VARS.raw", "auto-read-only": true, "discard": "unmap"}' \
-blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
-machine q35,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
-device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
-device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
-nodefaults \
-device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
-m 30720 \
-object '{"size": 32212254720, "mem-path": "/dev/shm", "share": true, "id": "mem-mem1", "qom-type": "memory-backend-file"}'  \
-smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
-numa node,memdev=mem-mem1,nodeid=0  \
-cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
-chardev socket,path=/var/tmp/avocado_arwp_g2r/monitor-qmpmonitor1-20240621-052512-EvVthgcD,id=qmp_id_qmpmonitor1,wait=off,server=on  \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,path=/var/tmp/avocado_arwp_g2r/monitor-catch_monitor-20240621-052512-EvVthgcD,id=qmp_id_catch_monitor,wait=off,server=on  \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device '{"ioport": 1285, "driver": "pvpanic", "id": "iduJuz5w"}' \
-chardev socket,path=/var/tmp/avocado_arwp_g2r/serial-serial0-20240621-052512-EvVthgcD,id=chardev_serial0,wait=off,server=on \
-device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
-chardev socket,id=seabioslog_id_20240621-052512-EvVthgcD,path=/var/tmp/avocado_arwp_g2r/seabios-20240621-052512-EvVthgcD,server=on,wait=off \
-device isa-debugcon,chardev=seabioslog_id_20240621-052512-EvVthgcD,iobase=0x402 \
-device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
-device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
-device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
-blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/timao/virtiofsd/source_mnt/rhel950-64-virtio-ovmf.qcow2", "cache": {"direct": true, "no-flush": false}}' \
-blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
-device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
-device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 0, "write-cache": "on", "bus": "pcie-root-port-2", "addr": "0x0"}' \
-chardev socket,id=char_virtiofs_fs,path=/home/timao/virtiofsd/source_virtiofsd.sock \
-device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
-device '{"id": "vufs_virtiofs_fs", "chardev": "char_virtiofs_fs", "tag": "myfs", "queue-size": 1024, "driver": "vhost-user-fs-pci", "bus": "pcie-root-port-3", "addr": "0x0"}' \
-chardev socket,id=char_virtiofs_fs2,path=/home/timao/virtiofsd/source_virtiofsd2.sock \
-device '{"id": "pcie-root-port-5", "port": 5, "driver": "pcie-root-port", "addr": "0x1.0x5", "bus": "pcie.0", "chassis": 6}' \
-device '{"id": "vufs_virtiofs_fs2", "chardev": "char_virtiofs_fs2", "tag": "myfs2", "queue-size": 1024, "driver": "vhost-user-fs-pci", "bus": "pcie-root-port-5", "addr": "0x0"}' \
-device '{"id": "pcie-root-port-4", "port": 6, "driver": "pcie-root-port", "addr": "0x1.0x6", "bus": "pcie.0", "chassis": 5}' \
-device '{"driver": "virtio-net-pci", "mac": "9a:b1:3c:cf:66:60", "id": "idiTi09X", "netdev": "idgL4k14", "bus": "pcie-root-port-4", "addr": "0x0"}' \
-netdev  '{"id": "idgL4k14", "type": "tap", "vhost": true}'  \
-vnc :1  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-monitor stdio \

4. On target node, create 2 directories the same as step1 and export them via virtiofsd

mkdir /home/timao/virtiofsd/local1/

mkdir /home/timao/virtiofsd/local2/

/usr/libexec/virtiofsd --shared-dir /home/timao/virtiofsd/local1/ --socket-path /home/timao/virtiofsd/tgt_virtiofsd.sock --cache auto &

/usr/libexec/virtiofsd --shared-dir /home/timao/virtiofsd/local2/ --socket-path /home/timao/virtiofsd/tgt_virtiofsd2.sock --cache auto &

Bootup a guest with the 2 shared directories on target node with '-incoming defer'

# /usr/libexec/qemu-kvm \
-S  \
-name 'avocado-vt-vm1'  \
-sandbox on,elevateprivileges=deny,obsolete=deny,resourcecontrol=deny \
-blockdev '{"node-name": "file_ovmf_code", "driver": "file", "filename": "/usr/share/OVMF/OVMF_CODE.secboot.fd", "auto-read-only": true, "discard": "unmap"}' \
-blockdev '{"node-name": "drive_ovmf_code", "driver": "raw", "read-only": true, "file": "file_ovmf_code"}' \
-blockdev '{"node-name": "file_ovmf_vars", "driver": "file", "filename": "/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel950-64-virtio-ovmf_qcow2_filesystem_VARS.raw", "auto-read-only": true, "discard": "unmap"}' \
-blockdev '{"node-name": "drive_ovmf_vars", "driver": "raw", "read-only": false, "file": "file_ovmf_vars"}' \
-machine q35,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
-device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
-device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
-nodefaults \
-device '{"driver": "VGA", "bus": "pcie.0", "addr": "0x2"}' \
-m 30720 \
-object '{"size": 32212254720, "mem-path": "/dev/shm", "share": true, "id": "mem-mem1", "qom-type": "memory-backend-file"}'  \
-smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
-numa node,memdev=mem-mem1,nodeid=0  \
-cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
-chardev socket,path=/var/tmp/avocado_arwp_g2r/monitor-qmpmonitor1-20240621-052512-EvVthgcD,id=qmp_id_qmpmonitor1,wait=off,server=on  \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,path=/var/tmp/avocado_arwp_g2r/monitor-catch_monitor-20240621-052512-EvVthgcD,id=qmp_id_catch_monitor,wait=off,server=on  \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device '{"ioport": 1285, "driver": "pvpanic", "id": "iduJuz5w"}' \
-chardev socket,path=/var/tmp/avocado_arwp_g2r/serial-serial0-20240621-052512-EvVthgcD,id=chardev_serial0,wait=off,server=on \
-device '{"id": "serial0", "driver": "isa-serial", "chardev": "chardev_serial0"}'  \
-chardev socket,id=seabioslog_id_20240621-052512-EvVthgcD,path=/var/tmp/avocado_arwp_g2r/seabios-20240621-052512-EvVthgcD,server=on,wait=off \
-device isa-debugcon,chardev=seabioslog_id_20240621-052512-EvVthgcD,iobase=0x402 \
-device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
-device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-1", "addr": "0x0"}' \
-device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
-blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/timao/virtiofsd/tgt_mnt/rhel950-64-virtio-ovmf.qcow2", "cache": {"direct": true, "no-flush": false}}' \
-blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
-device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
-device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 0, "write-cache": "on", "bus": "pcie-root-port-2", "addr": "0x0"}' \
-chardev socket,id=char_virtiofs_fs,path=/home/timao/virtiofsd/tgt_virtiofsd.sock \
-device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
-device '{"id": "vufs_virtiofs_fs", "chardev": "char_virtiofs_fs", "tag": "myfs", "queue-size": 1024, "driver": "vhost-user-fs-pci", "bus": "pcie-root-port-3", "addr": "0x0"}' \
-chardev socket,id=char_virtiofs_fs2,path=/home/timao/virtiofsd/tgt_virtiofsd2.sock \
-device '{"id": "pcie-root-port-5", "port": 5, "driver": "pcie-root-port", "addr": "0x1.0x5", "bus": "pcie.0", "chassis": 6}' \
-device '{"id": "vufs_virtiofs_fs2", "chardev": "char_virtiofs_fs2", "tag": "myfs2", "queue-size": 1024, "driver": "vhost-user-fs-pci", "bus": "pcie-root-port-5", "addr": "0x0"}' \
-device '{"id": "pcie-root-port-4", "port": 4, "driver": "pcie-root-port", "addr": "0x1.0x4", "bus": "pcie.0", "chassis": 5}' \
-device '{"driver": "virtio-net-pci", "mac": "9a:b1:3c:cf:66:60", "id": "idiTi09X", "netdev": "idgL4k14", "bus": "pcie-root-port-4", "addr": "0x0"}' \
-netdev  '{"id": "idgL4k14", "type": "tap", "vhost": true}'  \
-vnc :1  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot menu=off,order=cdn,once=c,strict=off \
-enable-kvm \
-monitor stdio \
-incoming defer

6. Migration

On target node:

(qemu) migrate_incoming tcp:[::]:5800

On souce node:

(qemu) migrate -d tcp:10.72.140.106:5800

Expected results

Migration success.

Actual results

Migration failed as below, and the VM keeps on running in source node.

On target node:

(qemu) qemu-kvm: Unknown ramblock "0000:00:01.6:00.0/virtio-net-pci.rom", cannot accept migration
qemu-kvm: error while loading state for instance 0x0 of device 'ram'
qemu-kvm: load of migration failed: Invalid argument
[2024-08-08T08:32:54Z INFO virtiofsd] Client disconnected, shutting down
[2024-08-08T08:32:54Z INFO virtiofsd] Client disconnected, shutting down
[1]- Done /usr/libexec/virtiofsd --shared-dir /home/timao/virtiofsd/local1/ --socket-path /home/timao/virtiofsd/tgt_virtiofsd.sock --cache auto
[2]+ Done /usr/libexec/virtiofsd --shared-dir /home/timao/virtiofsd/local2/ --socket-path /home/timao/virtiofsd/tgt_virtiofsd2.sock --cache auto

On source node:

(qemu) qemu-kvm: failed to save SaveStateEntry with id(name): 1(ram): -5
qemu-kvm: Unable to write to socket: Broken pipe

Note:

For the system disk, it's over a NFS backend.

Details

Description

What were you trying to do that didn't work?

Please provide the package NVR for which bug is seen:

How reproducible:

Steps to reproduce

Expected results

Actual results

Attachments

Easy Agile Planning Poker

Activity

People

Dates