Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-56945

[rhel10-beta][virtio-iommu][ice][hot-plug] virtio-iommu driver is not properly notified about the reserved regions

XMLWordPrintable

    • qemu-kvm-9.1.0-1.el10
    • None
    • Moderate
    • rhel-sst-virtualization
    • ssg_virtualization
    • 3
    • QE ack, Dev ack
    • False
    • Hide

      None

      Show
      None
    • None
    • Red Hat Enterprise Linux
    • None
    • x86_64
    • Linux
    • None

      What were you trying to do that didn't work?
      The qemu-kvm throws VFIO_MAP_DMA error when hot-plug 2 E810 PF into the VM has virtio-iommu device

      Please provide the package NVR for which bug is seen:
      Test env:
      qemu-kvm-9.0.0-7.el10.x86_64

      How reproducible:
      100%

      Steps to reproduce
      [1] start a VM with virtio+iommu device
      [2] hot-plug 2 ice PFs into the VM

      # virsh attach-device rhel10 /tmp/device/0000\:3b\:00.0.xml 
# virsh attach-device rhel10 /tmp/device/0000\:3b\:00.1.xml

      [3] check the VM dmesg

      [   46.478977] pci 0000:04:00.0: [8086:1592] type 00 class 0x020000 PCIe Endpoint
[   46.479564] pci 0000:04:00.0: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]
[   46.479930] pci 0000:04:00.0: BAR 3 [mem 0x00000000-0x0000ffff 64bit pref]
[   46.480283] pci 0000:04:00.0: ROM [mem 0x00000000-0x000fffff pref]
[   46.480496] pci 0000:04:00.0: Max Payload Size set to 128 (was 256, max 512)
[   46.483089] pci 0000:04:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:00:02.3 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[   46.484899] pci 0000:04:00.0: BAR 0 [mem 0x381800000000-0x381801ffffff 64bit pref]: assigned
[   46.485070] pci 0000:04:00.0: ROM [mem 0x82800000-0x828fffff pref]: assigned
[   46.485085] pci 0000:04:00.0: BAR 3 [mem 0x381802000000-0x38180200ffff 64bit pref]: assigned
[   46.509326] gnss: GNSS driver registered with major 239
[   46.603492] ice: Intel(R) Ethernet Connection E800 Series Linux Driver
[   46.603501] ice: Copyright (c) 2018, Intel Corporation.
[   46.603643] ice 0000:04:00.0: Adding to iommu group 22
[   46.714704] ice 0000:04:00.0: enabling device (0000 -> 0002)
[   47.198643] ice 0000:04:00.0: DDP package already present on device: ICE OS Default Package version 1.3.36.0
[   47.563638] ice 0000:04:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:00:02.3 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[   47.588522] ice 0000:04:00.0: PTP init successful
[   47.756508] ice 0000:04:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
[   47.757173] ice 0000:04:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
[   47.757383] ice 0000:04:00.0: Commit DCB Configuration to the hardware
[   47.772831] ice 0000:04:00.0 enp4s0np0: renamed from eth0
[   47.964357] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[   47.964367] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
[   48.737604] ice 0000:04:00.0: Using 64-bit DMA addresses
[   49.896298] ice 0000:04:00.0: IRDMA hardware initialization FAILED init_state=4 status=-110
[   50.050257] irdma ice.roce.1: probe with driver irdma failed with error -110
[   57.460477] pci 0000:05:00.0: [8086:1592] type 00 class 0x020000 PCIe Endpoint
[   57.461136] pci 0000:05:00.0: BAR 0 [mem 0x00000000-0x01ffffff 64bit pref]
[   57.461551] pci 0000:05:00.0: BAR 3 [mem 0x00000000-0x0000ffff 64bit pref]
[   57.461817] pci 0000:05:00.0: ROM [mem 0x00000000-0x000fffff pref]
[   57.461998] pci 0000:05:00.0: Max Payload Size set to 128 (was 256, max 512)
[   57.465559] pci 0000:05:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:00:02.4 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[   57.472155] pci 0000:05:00.0: BAR 0 [mem 0x382000000000-0x382001ffffff 64bit pref]: assigned
[   57.472753] pci 0000:05:00.0: ROM [mem 0x82600000-0x826fffff pref]: assigned
[   57.472768] pci 0000:05:00.0: BAR 3 [mem 0x382002000000-0x38200200ffff 64bit pref]: assigned
[   57.473395] ice 0000:05:00.0: Adding to iommu group 23
[   57.590319] ice 0000:05:00.0: enabling device (0000 -> 0002)
[   58.082219] ice 0000:05:00.0: DDP package already present on device: ICE OS Default Package version 1.3.36.0
[   58.314514] ice 0000:04:00.0 enp4s0np0: NIC Link is up 100 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg Advertised: On, Autoneg Negotiated: True, Flow Control: None
[   58.544625] ice 0000:05:00.0: 126.016 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:00:02.4 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
[   58.556146] ice 0000:05:00.0 enp5s0np1: renamed from eth0
[   58.884501] ice 0000:05:00.0: Using 64-bit DMA addresses
[   60.081772] ice 0000:05:00.0: IRDMA hardware initialization FAILED init_state=4 status=-110
*[   60.214655] irdma ice.roce.2: probe with driver irdma failed with error -110*
[   60.215637] ice 0000:05:00.0: PTP init successful
[   60.412954] ice 0000:05:00.0: DCB is enabled in the hardware, max number of TCs supported on this port are 8
[   60.412964] ice 0000:05:00.0: FW LLDP is disabled, DCBx/LLDP in SW mode.
[   60.413134] ice 0000:05:00.0: Commit DCB Configuration to the hardware
[   68.594941] ice 0000:05:00.0 enp5s0np1: NIC Link is up 100 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg Advertised: On, Autoneg Negotiated: True, Flow Control: None

      [4] check the qemu-kvm log

      # cat /var/log/libvirt/qemu/rhel10.log
2024-09-03T03:36:56.737898Z qemu-kvm: warning: virtio-iommu-memory-region-29-0: Notified about new host reserved regions after probe
2024-09-03T03:36:58.761690Z qemu-kvm: VFIO_MAP_DMA failed: Invalid argument
2024-09-03T03:36:58.761726Z qemu-kvm: vfio_container_dma_map(0x562ddf98c5d0, 0xffffffffffe00000, 0x200000, 0x7f51f0000000) = -22 (Invalid argument)
...
2024-09-03T03:39:28.900136Z qemu-kvm: VFIO_MAP_DMA failed: Invalid argument
2024-09-03T03:39:28.900173Z qemu-kvm: vfio_container_dma_map(0x562ddf98c5d0, 0xfffffffffffed000, 0x1000, 0x7f5143183000) = -22 (Invalid argument)
2024-09-03T03:42:02.166104Z qemu-kvm: terminating on signal 15 from pid 11678 (/usr/sbin/virtqemud)
2024-09-03 03:42:02.636+0000: shutting down, reason=shutdown

      Expected results
      No VFIO_MAP_DMA error

      Actual results
      The qemu-kvm throws VFIO_MAP_DMA error

              eauger Eric Auger
              yanghliu@redhat.com YangHang Liu
              virt-maint virt-maint
              YangHang Liu YangHang Liu
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: