Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-18189

QEMU with virtio-iommu enabled core dump after hot-plugged a vfio-pci device

XMLWordPrintable

    •  qemu-kvm-9.1.0-1.el9
    • None
    • Important
    • CustomerScenariosInitiative
    • rhel-sst-virtualization
    • ssg_virtualization
    • 5
    • QE ack
    • False
    • Hide

      None

      Show
      None
    • None
    • Red Hat Enterprise Linux
    • None
    • aarch64
    • None

      What were you trying to do that didn't work?

      Hotplug a vfio-pci device to the qemu instance, the qemu hit core dump.

      Please provide the package NVR for which bug is seen:

      host kernel: kernel-5.14.0-392.el9.aarch64+64k
      guest kernel: kernel-5.14.0-392.el9.aarch64+64k
      QEMU version: qemu-kvm-8.1.0-5.el9.aarch64
      edk2 version: edk2-aarch64-20230524-4.el9_3.noarch

      How reproducible:

      always

      Steps to reproduce

      1. Launch a guest without NICs and with virtio-iommu 
        MALLOC_PERTURB_=1  /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on \
    -blockdev '{"node-name": "file_aavmf_code", "driver": "file", "filename": "/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.qcow2", "auto-read-only": true, "discard": "unmap"}' \
    -blockdev '{"node-name": "drive_aavmf_code", "driver": "qcow2", "read-only": true, "file": "file_aavmf_code"}' \
    -blockdev '{"node-name": "file_aavmf_vars", "driver": "file", "filename": "/root/avocado/data/avocado-vt/avocado-vt-vm1_rhel940-aarch64-64k-virtio-scsi_qcow2_filesystem_VARS.qcow2", "auto-read-only": true, "discard": "unmap"}' \
    -blockdev '{"node-name": "drive_aavmf_vars", "driver": "qcow2", "read-only": false, "file": "file_aavmf_vars"}' \
    -machine virt,gic-version=host,pflash0=drive_aavmf_code,pflash1=drive_aavmf_vars,memory-backend=mem-machine_mem \
    -device '{"id": "pcie-root-port-0", "driver": "pcie-root-port", "multifunction": true, "bus": "pcie.0", "addr": "0x1", "chassis": 1}' \
    -device '{"id": "pcie-pci-bridge-0", "driver": "pcie-pci-bridge", "addr": "0x0", "bus": "pcie-root-port-0"}'  \
    -nodefaults \
    -device '{"driver": "virtio-iommu-pci", "bus": "pcie.0", "addr": "0x2"}' \
    -device '{"id": "pcie-root-port-1", "port": 1, "driver": "pcie-root-port", "addr": "0x1.0x1", "bus": "pcie.0", "chassis": 2}' \
    -device '{"driver": "virtio-gpu-pci", "bus": "pcie-root-port-1", "addr": "0x0"}' \
    -m 23552 \
    -object '{"size": 24696061952, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
    -smp 4,maxcpus=4,cores=2,threads=1,clusters=1,sockets=2  \
    -cpu 'host' \
    -serial unix:'/var/tmp/serial-serial0',server=on,wait=off \
    -device '{"id": "pcie-root-port-2", "port": 2, "driver": "pcie-root-port", "addr": "0x1.0x2", "bus": "pcie.0", "chassis": 3}' \
    -device '{"driver": "qemu-xhci", "id": "usb1", "bus": "pcie-root-port-2", "addr": "0x0"}' \
    -device '{"driver": "usb-tablet", "id": "usb-tablet1", "bus": "usb1.0", "port": "1"}' \
    -device '{"id": "pcie-root-port-3", "port": 3, "driver": "pcie-root-port", "addr": "0x1.0x3", "bus": "pcie.0", "chassis": 4}' \
    -device '{"id": "virtio_scsi_pci0", "driver": "virtio-scsi-pci", "bus": "pcie-root-port-3", "addr": "0x0"}' \
    -blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/rhel940-aarch64-64k-virtio-scsi.qcow2", "cache": {"direct": true, "no-flush": false}}' \
    -blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
    -device '{"driver": "scsi-hd", "id": "image1", "drive": "drive_image1", "write-cache": "on"}' \
    -device '{"id": "pcie-root-port-4", "port": 4, "driver": "pcie-root-port", "addr": "0x1.0x4", "bus": "pcie.0", "chassis": 5}' \
    -vnc :0  \
    -rtc base=utc,clock=host \
    -enable-kvm \
    -qmp stdio
      2. prepare the vfio device 
        # echo 1 > /sys/bus/pci/devices/0002\:01\:00.0/sriov_numvfs 
# ip link set eno1 vf 0 mac 92:61:97:2d:3a:20
# dpdk-devbind.py -b vfio-pci 0002:01:10.0
# driverctl list-devices network
0002:01:00.0 igb
0002:01:00.1 igb
0002:01:10.0 vfio-pci
      3. Hot-plug the vfio device into the guest 
        {"execute": "qmp_capabilities"}
{"return": {}}
{"execute": "device_add", "arguments": {"driver": "vfio-pci", "id": "hostdev1", "host": "0002:01:10.0", "bus": "pcie-root-port-4", "addr": "0x0"}}
{"return": {}}

      Expected results

      The vfio device can be hotplugged and can be found inside the guest.

      Actual results

      qemu-kvm: virtio-iommu virtio-iommu-memory-region-11-0 does not support frozen granule 0x1000
qemu: hardware error: vfio: DMA mapping failed, unable to continue
CPU #0:
 PC=ffff800008d09360 X00=0000000000000000 X01=ffff800009a60008
X02=ffff8005b22d0000 X03=ffff8005b22d0000 X04=ffff8005b22d0000
X05=4000000000000000 X06=00000002b2169ea8 X07=0000000100003ad3
X08=ffff800009f8d5d0 X09=ffff8000081c3db0 X10=00000000000010f0
X11=00000000ffff7fff X12=0000000000000000 X13=0000000000000000
X14=0000000000000000 X15=0000000000000000 X16=0000000000000000
X17=0000000000000000 X18=0000000000000000 X19=ffff800009a60008
X20=0000000000000000 X21=ffff800009f8c480 X22=ffff800009f796c0
X23=0000000000060000 X24=0000000000000000 X25=00000005f6604000
X26=00000000476edf58 X27=0000000000000000 X28=00000005edea0010
X29=ffff800009f4fd50 X30=ffff800008d0939c  SP=ffff800009f4fd50
PSTATE=604000c5 -ZC- EL1h     FPCR=00000000 FPSR=00000000
Q00=74726f7065696370:3d4547415353454d Q01=20203a342e31303a:30303a3030303020
Q02=6639303030387830:2d30303030303830 Q03=5d66657270207469:6234362066666666
Q04=5b20776f646e6977:2065676469726220 Q05=3030303030383030:30387830206d656d
Q06=cccccc03c0000000:cccccc03c0000000 Q07=636f6c206e6f6974:61746e656d656c70
Q08=0000000000000000:0000000000000000 Q09=0000000000000000:0000000000000000
Q10=0000000000000000:0000000000000000 Q11=0000000000000000:0000000000000000
Q12=0000000000000000:0000000000000000 Q13=0000000000000000:0000000000000000
Q14=0000000000000000:0000000000000000 Q15=0000000000000000:0000000000000000
Q16=0000000000000000:0000000000000000 Q17=0000000000000000:0000000000000000
Q18=0000000000000000:0000000000000000 Q19=0000000000000000:0000000000000000
Q20=0000000000000000:0000000000000000 Q21=0000000000000000:0000000000000000
Q22=0000000000000000:0000000000000000 Q23=0000000000000000:0000000000000000
Q24=0000000000000000:0000000000000000 Q25=0000000000000000:0000000000000000
Q26=0000000000000000:0000000000000000 Q27=0000000000000000:0000000000000000
Q28=0000000000000000:0000000000000000 Q29=0000000000000000:0000000000000000
Q30=0000000000000000:0000000000000000 Q31=0000000000000000:0000000000000000
CPU #1:
 PC=ffff8000087ab8d4 X00=ffff80000d9e3000 X01=0000000000000000
X02=ffff0000c9f71240 X03=0000000000000004 X04=0000000000000001
X05=0000000000000004 X06=ffff0000c0c3c000 X07=0000000000000001
X08=ffff000100c23140 X09=ffff8000087a7168 X10=0000000008090000
X11=0000000000010000 X12=0000060000000009 X13=0000000000000000
X14=0000000000000000 X15=ffff80000a592e2d X16=0000000000000000
X17=0000000000000000 X18=ffffffffffffffff X19=ffff0000c0066700
X20=ffff0000c007ac80 X21=ffff0000c0066700 X22=ffff0000c007ace8
X23=dead000000000122 X24=dead000000000100 X25=0000000000000001
X26=ffff800009244fc0 X27=ffff0000c35ebe48 X28=ffff800022a2fd20
X29=ffff800022a2f610 X30=ffff8000087a7168  SP=ffff800022a2f610
PSTATE=004000c5 ---- EL1h     FPCR=00000000 FPSR=00000000
Q00=e9b5dba5b5c0fbcf:71374491428a2f98 Q01=ab1c5ed5923f82a4:59f111f13956c25b
Q02=550c7dc3243185be:12835b01d807aa98 Q03=c19bf1749bdc06a7:80deb1fe72be5d74
Q04=240ca1cc0fc19dc6:efbe4786e49b69c1 Q05=76f988da5cb0a9dc:4a7484aa2de92c6f
Q06=bf597fc7b00327c8:a831c66d983e5152 Q07=1429296706ca6351:d5a79147c6e00bf3
Q08=53380d134d2c6dfc:2e1b213827b70a85 Q09=92722c8581c2c92e:766a0abb650a7354
Q10=c76c51a3c24b8b70:a81a664ba2bfe8a1 Q11=106aa070f40e3585:d6990624d192e819
Q12=34b0bcb52748774c:1e376c0819a4c116 Q13=682e6ff35b9cca4f:4ed8aa4a391c0cb3
Q14=8cc7020884c87814:78a5636f748f82ee Q15=c67178f2bef9a3f7:a4506ceb90befffa
Q16=d158c988cdd149ef:5bb8ec897c2ad079 Q17=fcc186c431ae02e5:692b933168d5b55e
Q18=7e5c00f755b92434:216e58dbf59af5d4 Q19=e2bb142d37f31a68:95d9ec2d92ea0749
Q20=23d817e11d0b4db9:40c7422daa9e1871 Q21=f7844e83bafd66a2:5e1f897673c3d4b6
Q22=0b2302ffda819c48:9a13bc4a6a2a78c2 Q23=a92c8d1ff6ecbe5f:3a2a591823a90743
Q24=f7efa3508148955d:6f330c9c1cb11b5a Q25=5f75de357e2cb52d:d5494b4f1db1d5c3
Q26=4a9dd2f18aa33f86:e2cc1b219b8a0ac8 Q27=0000000000000000:0000000000000000
Q28=0000000000000000:0000000000000000 Q29=0000000000000000:0000000000000000
Q30=0000000000000000:0000000000000000 Q31=0000000000000000:0000000000000000
CPU #2:
 PC=ffff800008d09360 X00=0000000000000002 X01=ffff800009a60008
X02=ffff8005b2410000 X03=ffff8005b2410000 X04=ffff8005b2410000
X05=4000000000000000 X06=00000002b2169ea8 X07=0000000000000003
X08=ffff0000c0683b50 X09=ffff8000081c3db0 X10=00000000000010f0
X11=7f7f7f7f7f7f7f7f X12=0101010101010101 X13=0000000000000020
X14=0000000000000000 X15=30387830206d656d X16=3030303030383030
X17=393030303878302d X18=ffffffffffffffff X19=ffff800009a60008
X20=0000000000000000 X21=ffff0000c0682a00 X22=ffff800009f796c0
X23=0000000000000000 X24=0000000000000000 X25=0000000000000000
X26=0000000000000000 X27=0000000000000000 X28=0000000000000000
X29=ffff80000b42fdd0 X30=ffff800008d0939c  SP=ffff80000b42fdd0
PSTATE=604000c5 -ZC- EL1h     FPCR=00000000 FPSR=00000000
Q00=0000fffffd877f80:0000fffffd877f90 Q01=ffffff8000000000:0000fffffd877f80
Q02=f00ff00ff00ff00f:f00ff00ff00ff00f Q03=fff0000000000000:fff0000000000000
Q04=f00ff00ff00ff00f:f00ff00ff00ff00f Q05=ff000f0f00000000:ff000f0f00000000
Q06=00000000656ec767:0000000000000008 Q07=0000000064e4aec5:0000000003938700
Q08=0000000000000000:0000000000000000 Q09=0000000000000000:0000000000000000
Q10=0000000000000000:0000000000000000 Q11=0000000000000000:0000000000000000
Q12=0000000000000000:0000000000000000 Q13=0000000000000000:0000000000000000
Q14=0000000000000000:0000000000000000 Q15=0000000000000000:0000000000000000
Q16=000000000000c000:0000000000000000 Q17=0000000000000015:0000010000000009
Q18=0000000000000000:0000000000000000 Q19=0000000000000000:0000000000000000
Q20=0000000000000000:0000000000000000 Q21=0000000000000000:0000000000000000
Q22=0000000000000000:0000000000000000 Q23=0000000000000000:0000000000000000
Q24=0000000000000000:0000000000000000 Q25=0000000000000000:0000000000000000
Q26=0000000000000000:0000000000000000 Q27=0000000000000000:0000000000000000
Q28=0000000000000000:0000000000000000 Q29=0000000000000000:0000000000000000
Q30=0000000000000000:0000000000000000 Q31=0000000000000000:0000000000000000
CPU #3:
 PC=ffff8000087f8d84 X00=0000000000000090 X01=0000000000000018
X02=ffff80000bec0018 X03=ffff80000a402840 X04=ffff80000a402840
X05=ffff0000c0bf3c0f X06=352e30353420205b X07=205d323435333835
X08=203030302e32203a X09=ffff8000087faef0 X10=3030303020696370
X11=302e30303a36303a X12=47203030302e3220 X13=6961766120732f62
X14=000000000000000a X15=ffff80008beefcc7 X16=3620666666663030
X17=6572702074696234 X18=ffffffffffffffff X19=ffff0000c0ff4080
X20=0000000000000032 X21=ffff0000c0ff4080 X22=ffff0000c0bf3cab
X23=0000000000000000 X24=00000000000000ab X25=ffff0000c0048100
X26=0000000000000000 X27=ffff80000a5856f0 X28=000000000000027f
X29=ffff80000beefc60 X30=ffff8000087faef0  SP=ffff80000beefc60
PSTATE=604000c5 -ZC- EL1h     FPCR=00000000 FPSR=00000010
Q00=0000000000000000:0000000000000000 Q01=0000000000000000:0000000000000007
Q02=0000000000000001:0000000000000004 Q03=0000000000000000:3fc015ad3b6200d1
Q04=00000000000f0000:00000000000f0000 Q05=65736f7072757073:79732f6d7368722f
Q06=0000ffffa449f730:0000ffffa6f03cf0 Q07=0000ffffa6f03db0:bd2cde126c7590ed
Q08=0000000000000000:0000000000000000 Q09=0000000000000000:0000000000000000
Q10=0000000000000000:0000000000000000 Q11=0000000000000000:0000000000000000
Q12=0000000000000000:0000000000000000 Q13=0000000000000000:0000000000000000
Q14=0000000000000000:0000000000000000 Q15=0000000000000000:0000000000000000
Q16=0000000000000000:0000000000000000 Q17=ffffffffffffffff:ffffffffffffffff
Q18=ffffffffffffffff:ffffffffffffffff Q19=ffff0000ffff0000:ffff0000ffff0000
Q20=0000000000000000:0000000000000001 Q21=0000000000000000:0000000000000001
Q22=0000000000000000:0000000000000001 Q23=0000000100000001:0000000100000001
Q24=ffffff02ffffff00:ffffff01ffffff00 Q25=ffff0002ffff0000:ffff0001ffff0000
Q26=0000000100000001:0000000100000001 Q27=0000000100000001:0000000100000001
Q28=0000000000000005:0000000300000002 Q29=0000000000000000:0000000000000003
Q30=0000000000000000:0000000000000005 Q31=0000000000000000:0000000000000000
Aborted (core dumped)

      Guest serial log:

      [  450.428556] pcieport 0000:00:01.4: pciehp: Slot(0-4): Button press: will power on in 5 sec
[  450.428681] pcieport 0000:00:01.4: pciehp: Slot(0-4): Card present
[  450.428685] pcieport 0000:00:01.4: pciehp: Slot(0-4): Link Up
[  450.582812] pci 0000:06:00.0: [8086:1520] type 00 class 0x020000
[  450.583005] pci 0000:06:00.0: reg 0x10: [mem 0x00000000-0x0000ffff 64bit pref]
[  450.583141] pci 0000:06:00.0: reg 0x1c: [mem 0x00000000-0x0000ffff 64bit pref]
[  450.583542] pci 0000:06:00.0: 2.000 Gb/s available PCIe bandwidth, limited by

              eauger Eric Auger
              rhn-support-yihyu Yihuang Yu
              virt-maint virt-maint
              Yingshun Cui Yingshun Cui
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

                Created:
                Updated: