Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-3293

Test Coverage: [openvswitch] RCU CPU stall in ovs_dp_masks_rebalance during OVS-DPDK rebalance

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      ( ) The test coverage is aligned with the epic's acceptance criteria

      Show
      ( ) The test coverage is aligned with the epic's acceptance criteria
    • rhel-10
    • None

      This task is tracking the test case writing activities to cover the bug described below.

       Problem Description: Clearly explain the issue.

      Guest VM hangs during OVS-DPDK rebalance due to RCU stall in openvswitch kernel module
       

       Impact Assessment: Describe the severity and impact (e.g., network down,availability of a workaround, etc.).

       

       Software Versions: Specify the exact versions in use (e.g.,openvswitch3.1-3.1.0-147.el8fdp).

      kernel-6.12.0-124.38.1.el10_1
      openvswitch3.5-3.5.2-63.el10fdp.x86_64.rpm
      openvswitch3.6-3.6.1-11.el10fdp.x86_64.rpm 

        Issue Type: Indicate whether this is a new issue or a regression (if a regression, state the last known working version).

       

       Reproducibility: Confirm if the issue can be reproduced consistently. If not, describe how often it occurs.

      It is not consistently reproducible. This issue has occurred twice so far in different test environments.
       

       Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

      Run ovs dpdk rebalance jobs
      rhel 10.1 ovs3.5 i40e job:
      https://beaker.engineering.redhat.com/jobs/12335212
      rhel 10.1 ovs3.6 cx5 job:
      https://beaker.engineering.redhat.com/jobs/12336186

      For the i40e job steps:
      Running all-ovs-dpdk-multiple-queue-auto-rebalance-4pmds-2queues-test-with-4vms-test

      for i40e job, 
systemctl start openvswitch
ovs-vsctl set Open_vSwitch . other_config={}
systemctl restart openvswitch
ovs-vsctl list-br | xargs -I {} ovs-vsctl del-br {}
systemctl stop openvswitch
rm -rf /dev/hugepages/*
ip link set ens5f0np0 down
ip link set ens5f1np1 down
nmcli dev set ens5f0np0 managed no
nmcli dev set ens5f1np1 managed no
driverctl -v set-override 0000:a0:00.0 vfio-pci
driverctl -v set-override 0000:a0:00.1 vfio-pci
modprobe vfio-pci
modprobe vfio
systemctl start openvswitch
ovs-vsctl --if-exists del-br ovsbr0
ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true"
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,4096"
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x280000000000000028
ovs-vsctl --no-wait set Open_vSwitch . other_config:vhost-iommu-support=true
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-rebal-interval=1
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-load-threshold=50
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-improvement-threshold=10
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb=true
ovs-vsctl set Open_vSwitch . other_config:pmd-rxq-assign="cycles"
"ovs-vsctl add-br ovsbr0
ovs-vsctl set bridge ovsbr0 datapath_type=netdev
ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk type=dpdk options:dpdk-devargs=0000:a0:00.0 options:n_rxq=4 ofport_request=1 mtu_request=1500
ovs-vsctl add-port ovsbr0 dpdk1 -- set Interface dpdk1 type=dpdk type=dpdk options:dpdk-devargs=0000:a0:00.1 options:n_rxq=4 ofport_request=2 mtu_request=1500
    for i in `seq 0 7`
    do
        ovs-vsctl --if-exists del-port ovsbr0 vhost${i}
        ovs-vsctl add-port ovsbr0 vhost${i} -- set interface vhost${i} type=dpdkvhostuserclient options:vhost-server-path=/tmp/vhost${i} ofport_request=$((3 + i)) mtu_request=1500
    done
ovs-ofctl add-flow ovsbr0 in_port=dpdk0,idle_timeout=0,action=output:vhost2
ovs-ofctl add-flow ovsbr0 in_port=vhost2,idle_timeout=0,action=output:dpdk0
ovs-ofctl add-flow ovsbr0 in_port=dpdk1,idle_timeout=0,action=output:vhost0,vhost1,vhost3,vhost4,vhost5,vhost6,vhost7
ovs-ofctl add-flow ovsbr0 in_port=vhost0,idle_timeout=0,action=output:dpdk1
ovs-ofctl add-flow ovsbr0 in_port=vhost1,idle_timeout=0,action=output:dpdk1
ovs-ofctl add-flow ovsbr0 in_port=vhost3,idle_timeout=0,action=output:dpdk1
ovs-ofctl add-flow ovsbr0 in_port=vhost4,idle_timeout=0,action=output:dpdk1
ovs-ofctl add-flow ovsbr0 in_port=vhost5,idle_timeout=0,action=output:dpdk1
ovs-ofctl add-flow ovsbr0 in_port=vhost6,idle_timeout=0,action=output:dpdk1
ovs-ofctl add-flow ovsbr0 in_port=vhost7,idle_timeout=0,action=output:dpdk1
ovs-appctl dpif-netdev/pmd-rxq-show
Displaying last 60 seconds pmd usage %
pmd thread numa_id 1 core_id 3:
  isolated : false
  port: dpdk0             queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  port: vhost3            queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  port: vhost4            queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  overhead: NOT AVAIL
pmd thread numa_id 1 core_id 5:
  isolated : false
  port: dpdk0             queue-id:  1 (enabled)   pmd usage: NOT AVAIL
  port: vhost2            queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  port: vhost5            queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  overhead: NOT AVAIL
pmd thread numa_id 1 core_id 67:
  isolated : false
  port: dpdk1             queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  port: vhost1            queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  port: vhost6            queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  overhead: NOT AVAIL
pmd thread numa_id 1 core_id 69:
  isolated : false
  port: dpdk1             queue-id:  1 (enabled)   pmd usage: NOT AVAIL
  port: vhost0            queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  port: vhost7            queue-id:  0 (enabled)   pmd usage: NOT AVAIL
  overhead: NOT AVAIL
virsh define vm-1
LIBGUESTFS_BACKEND=direct
virt-customize -d gmp-1     --root-password password:redhat     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-config.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-final.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init-local.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init.service'     --firstboot-command 'reboot'
systemctl list-units --state=stop --type=service | grep libvirtd || systemctl restart libvirtd
systemctl start libvirtd.service
systemctl start virtqemud.socket
virsh dumpxml gmp-1
<domain type='kvm' id='1'>
  <name>gmp-1</name>
  <uuid>2466d1a6-c9ac-4c66-855a-71701cde34d0</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='7'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='71'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel10.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp//gmp-1.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='52:54:00:11:8f:ea'/>
      <source type='unix' path='/tmp/vhost0' mode='server'/>
      <target dev='vhost0'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='52:54:00:11:8f:eb'/>
      <source type='unix' path='/tmp/vhost1' mode='server'/>
      <target dev='vhost1'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='virtio' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
      <alias name='iommu0'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c412,c835</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c412,c835</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+984</label>
    <imagelabel>+107:+984</imagelabel>
  </seclabel>
</domain>
virsh start gmp-1
sleep 60
virsh dumpxml gmp-1
inside gmp-1:
driverctl -v set-override 0000:03:00.0 vfio-pci
driverctl -v set-override 0000:04:00.0 vfio-pci
dpdk-testpmd -l 0,1,2      --socket-mem 1024      --legacy-mem      -n 4     --     --burst=64     --forward-mode=mac     --port-topology=paired          --disable-rss     -i     --rxq=1     --txq=1     --rxd=1024     --txd=1024     --nb-cores=2     --max-pkt-len=1500          --auto-start
virsh define vm-2
LIBGUESTFS_BACKEND=direct
virt-customize -d gmp-2     --root-password password:redhat     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-config.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-final.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init-local.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init.service'     --firstboot-command 'reboot'
systemctl list-units --state=stop --type=service | grep libvirtd || systemctl restart libvirtd
systemctl start libvirtd.service
systemctl start virtqemud.socket
virsh dumpxml gmp-2
<domain type='kvm'>
  <name>gmp-2</name>
  <uuid>d7ae8b2f-4ffc-4a98-bced-eaa2c0c6e109</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='11'/>
    <vcpupin vcpu='1' cpuset='75'/>
    <vcpupin vcpu='2' cpuset='73'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel10.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp//gmp-2.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'/>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='58:11:58:3c:20:46'/>
      <source type='unix' path='/tmp/vhost2' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='58:11:58:84:09:a6'/>
      <source type='unix' path='/tmp/vhost3' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='virtio' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>
inside guest:
driverctl -v set-override 0000:03:00.0 vfio-pci
driverctl -v set-override 0000:04:00.0 vfio-pci
dpdk-testpmd -l 0,1,2      --socket-mem 1024      --legacy-mem      -n 4     --     --burst=64     --forward-mode=mac     --port-topology=paired          --disable-rss     -i     --rxq=1     --txq=1     --rxd=1024     --txd=1024     --nb-cores=2     --max-pkt-len=1500          --auto-start
virsh define vm-3
LIBGUESTFS_BACKEND=direct
virt-customize -d gmp-3     --root-password password:redhat     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-config.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-final.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init-local.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init.service'     --firstboot-command 'reboot'
systemctl start libvirtd.service
systemctl start virtqemud.socket
virsh dumpxml gmp-3
<domain type='kvm'>
  <name>gmp-3</name>
  <uuid>9c73a791-888f-41a7-803a-c7d01bdb3f4d</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='13'/>
    <vcpupin vcpu='1' cpuset='15'/>
    <vcpupin vcpu='2' cpuset='77'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel10.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp//gmp-3.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'/>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='58:11:58:86:ae:c3'/>
      <source type='unix' path='/tmp/vhost4' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='58:11:58:86:5c:d0'/>
      <source type='unix' path='/tmp/vhost5' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='virtio' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>
virsh start gmp-3
sleep 60
virsh dumpxml gmp-3
inside guest:
driverctl -v set-override 0000:03:00.0 vfio-pci
driverctl -v set-override 0000:04:00.0 vfio-pci
dpdk-testpmd -l 0,1,2      --socket-mem 1024      --legacy-mem      -n 4     --     --burst=64     --forward-mode=mac     --port-topology=paired          --disable-rss     -i     --rxq=1     --txq=1     --rxd=1024     --txd=1024     --nb-cores=2     --max-pkt-len=1500          --auto-start
virsh define vm-4
LIBGUESTFS_BACKEND=direct
virt-customize -d gmp-4     --root-password password:redhat     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-config.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-final.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init-local.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init.service'     --firstboot-command 'reboot'
systemctl list-units --state=stop --type=service | grep libvirtd || systemctl restart libvirtd
systemctl start libvirtd.service
virsh dumpxml gmp-4
<domain type='kvm'>
  <name>gmp-4</name>
  <uuid>ad316ef1-5242-4e26-a085-62241c35cdb5</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='17'/>
    <vcpupin vcpu='1' cpuset='81'/>
    <vcpupin vcpu='2' cpuset='79'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel10.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='4194304' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp//gmp-4.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'/>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='58:11:58:2a:b7:a7'/>
      <source type='unix' path='/tmp/vhost6' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='58:11:58:a5:7b:f6'/>
      <source type='unix' path='/tmp/vhost7' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='virtio' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>
virsh start gmp-4
sleep 60
virsh dumpxml gmp-4
The system is hung and has call trace.

       
      For the cx5 job steps:

      for i40e job, running  ovs_dpdk_queue_rebalance-core_num-2-without-mac-learning--with-emc--setup-environment case
systemctl start openvswitch
ovs-vsctl set Open_vSwitch . other_config={}
systemctl restart openvswitch
ovs-vsctl list-br | xargs -I {} ovs-vsctl del-br {}
systemctl stop openvswitch
rm -rf /dev/hugepages/*
ip link set ens7f0np0 down
ip link set ens7f1np1 down
nmcli dev set ens7f0np0 managed no
nmcli dev set ens7f1np1 managed no
modprobe vfio-pci
modprobe vfio
systemctl start openvswitch
ovs-vsctl --if-exists del-br ovsbr0
ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true"
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="4096,4096
ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x28
ovs-vsctl --no-wait set Open_vSwitch . other_config:vhost-iommu-support=true
ovs-vsctl --no-wait set Open_vSwitch . other_config:emc-insert-inv-prob=1
ovs-vsctl add-br ovsbr0
ovs-vsctl add-port ovsbr0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:a0:1e:54" options:n_rxq=1 mtu_request=1500 
ovs-vsctl add-port ovsbr0 dpdk1 -- set Interface dpdk1 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:a0:1e:55" options:n_rxq=1 mtu_request=1500
ovs-vsctl add-port ovsbr0 vhost0 -- set interface vhost0 type=dpdkvhostuserclient options:vhost-server-path=/tmp/vhost0 mtu_request=1500
ovs-vsctl add-port ovsbr0 vhost1 -- set interface vhost1 type=dpdkvhostuserclient options:vhost-server-path=/tmp/vhost1 mtu_request=1500
ovs-vsctl set Interface dpdk0  ofport_request=1
ovs-vsctl set Interface dpdk1  ofport_request=2
ovs-vsctl set Interface vhost0 ofport_request=3
ovs-vsctl set Interface vhost1 ofport_request=4
ovs-ofctl del-flows ovsbr0
/usr/bin/ovs-ofctl -O OpenFlow13 --timeout 10 add-flow ovsbr0 idle_timeout=0,in_port=1,action=output:3
/usr/bin/ovs-ofctl -O OpenFlow13 --timeout 10 add-flow ovsbr0 idle_timeout=0,in_port=3,action=output:1
/usr/bin/ovs-ofctl -O OpenFlow13 --timeout 10 add-flow ovsbr0 idle_timeout=0,in_port=2,action=output:4
/usr/bin/ovs-ofctl -O OpenFlow13 --timeout 10 add-flow ovsbr0 idle_timeout=0,in_port=4,action=output:2
ovs-ofctl add-flow ovsbr0 actions=NORMAL
ovs-vsctl set Interface dpdk0 other_config:pmd-rxq-affinity="0:3"     -- set Interface vhost0 other_config:pmd-rxq-affinity="0:3"     -- set Interface dpdk1 other_config:pmd-rxq-affinity="0:5"     -- set Interface vhost1 other_config:pmd-rxq-affinity="0:5"
ovs-vsctl set Interface dpdk0 other_config:pmd-rxq-affinity="0:3"     -- set Interface vhost0 other_config:pmd-rxq-affinity="0:3"     -- set Interface dpdk1 other_config:pmd-rxq-affinity="0:5"     -- set Interface vhost1 other_config:pmd-rxq-affinity="0:5"
virsh define g1.xml
LIBGUESTFS_BACKEND=direct
virt-customize -d gg     --root-password password:redhat     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-config.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-final.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init-local.service'     --firstboot-command 'rm /etc/systemd/system/multi-user.target.wants/cloud-init.service'     --firstboot-command 'reboot'
systemctl list-units --state=stop --type=service | grep libvirtd || systemctl restart libvirtd
systemctl start libvirtd.service
systemctl start virtqemud.socket
virsh start gg
sleep 60
virsh dumpxml gg
<domain type='kvm' id='1'>
  <name>gg</name>
  <uuid>37425e76-af6a-44a6-aba0-73434afe34c0</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='9'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='71'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel10.0.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/tmp//gg.qcow2' index='1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:bb:63:7b'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='52:54:00:11:8f:ea'/>
      <source type='unix' path='/tmp/vhost0' mode='server'/>
      <target dev='vhost0'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='52:54:00:11:8f:eb'/>
      <source type='unix' path='/tmp/vhost1' mode='server'/>
      <target dev='vhost1'/>
      <model type='virtio'/>
      <driver name='vhost' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='virtio' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <watchdog model='itco' action='reset'>
      <alias name='watchdog0'/>
    </watchdog>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
      <alias name='iommu0'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c135,c422</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c135,c422</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+984</label>
    <imagelabel>+107:+984</imagelabel>
  </seclabel>
</domain>
The system is hung and has call trace.

       Expected Behavior: Describe what should happen under normal circumstances.

      The guest works well when running ovs dpdk rebalance jobs.
       

       Observed Behavior: Explain what actually happens.

      The guest vm hangs during OVS-DPDK rebalance due to RCU stall in openvswitch kernel module
      The jurnalctl log of job 12335212.
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2026/02/123352/12335212/20701058/212798077/journalctl 

      Feb 18 01:49:28 dell-per760-11.rhts.eng.pek2.redhat.com kernel: 
Feb 18 01:49:28 dell-per760-11.rhts.eng.pek2.redhat.com kernel: Showing busy workqueues and worker pools:
Feb 18 01:49:28 dell-per760-11.rhts.eng.pek2.redhat.com kernel: workqueue events: flags=0x0
Feb 18 01:49:28 dell-per760-11.rhts.eng.pek2.redhat.com kernel:   pwq 122: cpus=30 node=0 flags=0x0 nice=0 active=1 refcnt=2
Feb 18 01:49:28 dell-per760-11.rhts.eng.pek2.redhat.com kernel:     pending: ovs_dp_masks_rebalance [openvswitch]
Feb 18 01:49:28 dell-per760-11.rhts.eng.pek2.redhat.com kernel: workqueue mm_percpu_wq: flags=0x8
Feb 18 01:49:28 dell-per760-11.rhts.eng.pek2.redhat.com kernel:   pwq 122: cpus=30 node=0 flags=0x0 nice=0 active=1 refcnt=2
Feb 18 01:49:28 dell-per760-11.rhts.eng.pek2.redhat.com kernel:     pending: vmstat_update
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: rcu:         102-...!: (1 GPs behind) idle=6c2c/1/0x4000000000000002 softirq=56968/56968 fqs=4
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: rcu:         (detected by 102, t=214582 jiffies, g=1228313, q=58072 ncpus=128)
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: CPU: 102 UID: 0 PID: 0 Comm: swapper/102 Kdump: loaded Not tainted 6.12.0-124.38.1.el10_1.x86_64 #1 PREEMPT(voluntary) 
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: Hardware name: Dell Inc. PowerEdge R760/0024FG, BIOS 2.1.5 03/14/2024
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: RIP: 0010:cpuidle_enter_state+0xc0/0x410
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: Code: d3 01 00 00 e8 91 7b 39 ff e8 1c f0 ff ff 49 89 c5 0f 1f 44 00 00 31 ff e8 2d ed 37 ff 45 84 ff 0f 85 a5 01 00 00 fb 45 85 f6 <0f> 88 84 01 00 00 49 63 d6 48 8d 04 52 48 8d 04 82 49 8d 0c c4 48
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: RSP: 0018:ff7d4c3746987e70 EFLAGS: 00000202
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: RAX: ff370294bfec0000 RBX: 0000000000000001 RCX: 0000000000000000
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: RDX: 000027742ece0797 RSI: 00002796ccc794e7 RDI: 0000000000000000
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: RBP: ffaf4c27c0cc2a60 R08: 0000000000000000 R09: 0000000000000000
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: R10: 00000000ffffffff R11: ff370294bfef06ec R12: ffffffffb98d6260
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: R13: 000027742ece0797 R14: 0000000000000001 R15: 0000000000000000
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: FS:  0000000000000000(0000) GS:ff370294bfec0000(0000) knlGS:0000000000000000
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: CR2: 000055f54665f018 CR3: 000000027a92a004 CR4: 0000000000f73ef0
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: PKRU: 55555554
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel: Call Trace:
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  <IRQ>
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? show_trace_log_lvl+0x1b0/0x2f0
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? show_trace_log_lvl+0x1b0/0x2f0
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? cpuidle_enter+0x2d/0x40
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? rcu_dump_cpu_stacks+0xac/0xed
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? print_other_cpu_stall.cold+0x3c/0x1cf
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? check_cpu_stall+0x1bf/0x220
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? rcu_pending+0x2f/0x220
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? rcu_sched_clock_irq+0xa8/0x370
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? update_process_times+0x70/0xb0
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? tick_nohz_handler+0x8f/0x140
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? __pfx_tick_nohz_handler+0x10/0x10
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? __hrtimer_run_queues+0x139/0x2a0
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? hrtimer_interrupt+0xfa/0x210
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? __sysvec_apic_timer_interrupt+0x52/0x100
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? sysvec_apic_timer_interrupt+0x6c/0x90
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  </IRQ>
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  <TASK>
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  ? cpuidle_enter_state+0xc0/0x410
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  cpuidle_enter+0x2d/0x40
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  cpuidle_idle_call+0xef/0x150
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  do_idle+0x73/0xd0
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  cpu_startup_entry+0x29/0x30
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  start_secondary+0x119/0x140
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  common_startup_64+0x13e/0x141
Feb 18 01:49:29 dell-per760-11.rhts.eng.pek2.redhat.com kernel:  </TASK>

      The jurnalctl log of job 12336186
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2026/02/123361/12336186/20702875/212821113/journalctl

      Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: lowmem_reserve[]: 0 0 0 0 0
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Node 0 DMA32: 5*4kB (UM) 4*8kB (UM) 3*16kB (M) 4*32kB (UM) 3*64kB (UM) 3*128kB (UM) 3*256kB (M) 5*512kB (UM) 3*1024kB (M) 3*2048kB (M) 270*4096kB (M) = 1119268kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Node 0 Normal: 36387*4kB (UME) 32282*8kB (UME) 15851*16kB (UME) 7082*32kB (UME) 3795*64kB (UME) 1546*128kB (UME) 649*256kB (UME) 260*512kB (UME) 100*1024kB (UME) 44*2048kB (UME) 7533*4096kB (UM) = 32671756kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Node 1 Normal: 10560*4kB (UME) 3126*8kB (UME) 3693*16kB (UME) 927*32kB (UME) 645*64kB (UME) 241*128kB (UME) 207*256kB (UME) 72*512kB (UM) 60*1024kB (UM) 21*2048kB (UME) 8719*4096kB (UM) = 36135456kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Node 0 hugepages_total=24 hugepages_free=20 hugepages_surp=0 hugepages_size=1048576kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Node 1 hugepages_total=24 hugepages_free=12 hugepages_surp=0 hugepages_size=1048576kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: 1374839 total pagecache pages
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: 0 pages in swap cache
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Free swap  = 33554428kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Total swap = 33554428kB
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: 33397137 pages RAM
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: 0 pages HighMem/MovableOnly
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: 1151463 pages reserved
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: 0 pages cma reserved
Feb 18 06:31:03 dell-per760-09.rhts.eng.pek2.redhat.com kernel: 0 pages hwpoisoned
Feb 18 06:31:05 dell-per760-09.rhts.eng.pek2.redhat.com unknown[103790]: List of m Tasks: Stop
Feb 18 06:31:07 dell-per760-09.rhts.eng.pek2.redhat.com logger[103792]: List of t Tasks: Start
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: sysrq: Show State
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: task:systemd         state:S stack:0     pid:1     tgid:1     ppid:0      task_flags:0x400100 flags:0x00000002
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: Call Trace:
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  <TASK>
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  __schedule+0x2aa/0x660
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  schedule+0x27/0xa0
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  schedule_hrtimeout_range_clock+0x1bf/0x1d0
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ep_poll+0x3bc/0x410
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? __pfx_ep_autoremove_wake_function+0x10/0x10
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  do_epoll_wait+0xc1/0xe0
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  __x64_sys_epoll_wait+0x71/0x110
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  do_syscall_64+0x7d/0x160
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? pidfd_show_fdinfo+0x86/0x100
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? from_kgid_munged+0x12/0x30
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? cp_new_stat+0x131/0x170
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? __ct_user_enter+0x29/0xd0
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? syscall_exit_to_user_mode+0xe3/0x190
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? do_syscall_64+0x89/0x160
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? timerqueue_add+0x71/0xc0
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? enqueue_hrtimer+0x38/0x90
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? hrtimer_start_range_ns+0x73/0xc0
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? do_timerfd_settime+0x2c5/0x500
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? __x64_sys_timerfd_settime+0xa5/0x100
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? __ct_user_enter+0x29/0xd0
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? syscall_exit_to_user_mode+0xe3/0x190
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? do_syscall_64+0x89/0x160
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? __ct_user_enter+0x29/0xd0
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? syscall_exit_to_user_mode+0xe3/0x190
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? do_syscall_64+0x89/0x160
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? __ct_user_enter+0x29/0xd0
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? syscall_exit_to_user_mode+0xe3/0x190
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  ? do_syscall_64+0x89/0x160
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: RIP: 0033:0x7f23e912def7
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: RSP: 002b:00007ffeccd96828 EFLAGS: 00000202 ORIG_RAX: 00000000000000e8
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: RAX: ffffffffffffffda RBX: 00000000000000b8 RCX: 00007f23e912def7
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: RDX: 00000000000000b8 RSI: 00005644dfe14c40 RDI: 0000000000000004
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: RBP: 00007ffeccd96950 R08: de7539b50fa488ff R09: 0000000000000004
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: R10: 00000000ffffffff R11: 0000000000000202 R12: 00005644dfe14c40
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel: R13: 00005644dfcff7e0 R14: 0000000000000055 R15: ffffffffffffffff
Feb 18 06:31:08 dell-per760-09.rhts.eng.pek2.redhat.com kernel:  </TASK>

       Troubleshooting Actions: Outline the steps taken to diagnose or resolve the issue so far.

       

       Logs: If you collected logs please provide them (e.g. sos report, /var/log/openvswitch/* , testpmd console)

      console log:
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2026/02/123352/12335212/20701058/console.log
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2026/02/123361/12336186/20702875/console.log
      call trace on console log:

              ovs-qe Openvswtich Quality Engineering Bot
              tli@redhat.com Ting Li
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: