Description of problem:

Version-Release number of selected component (if applicable):
kernel 5.14.0-284.60.1.el9_2.x86_64
ovn23.09-central-23.09.0-139.el9fdp
openvswitch3.2-3.2.0-63.el9fdp

How reproducible:

Steps to Reproduce:
Run ovs dpdk vhostuser with ovn performance case
for example 1q case:
1.build following ovs bridge
Bridge br-int
fail_mode: secure
datapath_type: netdev
Port vhost1
Interface vhost1
type: dpdkvhostuserclient
options:

{vhost-server-path="/tmp/vhostuser/vhost1"}

Port vhost0
Interface vhost0
type: dpdkvhostuserclient
options:

{vhost-server-path="/tmp/vhostuser/vhost0"}

Port dpdk0
Interface dpdk0
type: dpdk
options:

{dpdk-devargs="0000:07:00.0", n_rxq="1", n_rxq_desc="2048", n_txq_desc="2048"}

Port br-int
Interface br-int
type: internal
Port dpdk1
Interface dpdk1
type: dpdk
options:

{dpdk-devargs="0000:07:00.1", n_rxq="1", n_rxq_desc="2048", n_txq_desc="2048"}

ovs_version: "3.2.3"

2. run following ovn configuration
ovn-nbctl ls-add ls1
ovn-nbctl ls-add ls2
ovn-nbctl lsp-add ls1 vm1
ovn-nbctl lsp-set-addresses vm1 00:00:00:00:00:01 "unknown"
ovn-nbctl lsp-add ls1 phy1
ovn-nbctl lsp-set-addresses phy1 00:00:00:00:00:02 "unknown"
ovn-nbctl lsp-add ls2 vm2
ovn-nbctl lsp-set-addresses vm2 00:00:00:00:00:03 "unknown"
ovn-nbctl lsp-add ls2 phy2
ovn-nbctl lsp-set-addresses phy2 00:00:00:00:00:04 "unknown"
ovs-vsctl set interface dpdk0 external_ids:iface-id=phy1
ovs-vsctl set interface dpdk1 external_ids:iface-id=phy2
ovs-vsctl set interface vhost0 external_ids:iface-id=vm1
ovs-vsctl set interface vhost1 external_ids:iface-id=vm2
ovn-nbctl set logical_switch ls1 other_config:vlan-passthru=true
ovn-nbctl set logical_switch ls2 other_config:vlan-passthru=true
systemctl restart ovn-controller
3.start guest with following xml
<network>
<name>default</name>
<uuid>3076b7a7-d2af-44c4-b4e1-7e4e9648465b</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr0' stp='on' delay='0'/>
<mac address='52:54:00:04:4b:b8'/>
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254'/>
<host mac='52:54:00:01:02:03' ip='192.168.122.123'/>
</dhcp>
</ip>
</network>

Create g1.xml
<domain type='kvm'>
<name>g1</name>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB'/>
</hugepages>
<locked/>
<access mode='shared'/>
</memoryBacking>
<vcpu placement='static'>3</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='4'/>
<vcpupin vcpu='1' cpuset='30'/>
<vcpupin vcpu='2' cpuset='2'/>
<emulatorpin cpuset='0,28'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
</numatune>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='q35'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pmu state='off'/>
<vmport state='off'/>
<ioapic driver='qemu'/>
</features>
<cpu mode='host-passthrough' check='none'>
<feature policy='require' name='tsc-deadline'/>
<numa>
<cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
</numa>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/g1.qcow2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</disk>
<controller type='usb' index='0' model='none'>
<alias name='usb'/>
</controller>
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x8'/>
<alias name='pci.3'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x9'/>
<alias name='pci.4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0xa'/>
<alias name='pci.5'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0xb'/>
<alias name='pci.6'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</controller>
<controller type='sata' index='0'>
<alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:01:02:03'/>
<source bridge='virbr0'/>
<model type='virtio'/>
</interface>
<interface type='vhostuser'>
<mac address='00:de:ad:00:00:01'/>
<source type='unix' path='/tmp/vhostuser/vhost0' mode='server'/>
<model type='virtio'/>
<driver name='vhost' queues='1' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'>
<host mrg_rxbuf='off'/>
</driver>
<address type='pci' domain='0x0000' bus='0x3' slot='0x00' function='0x0'/>
</interface>
<interface type='vhostuser'>
<mac address='00:de:ad:00:00:02'/>
<source type='unix' path='/tmp/vhostuser/vhost1' mode='server'/>
<model type='virtio'/>
<driver name='vhost' queues='1' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'>
<host mrg_rxbuf='off'/>
</driver>
<address type='pci' domain='0x0000' bus='0x4' slot='0x00' function='0x0'/>
</interface>
<serial type='pty'>
<source path='/dev/pts/1'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
<alias name='serial0'/>
</serial>
<console type='pty' tty='/dev/pts/1'>
<source path='/dev/pts/1'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
</console>
<input type='mouse' bus='ps2'>
<alias name='input0'/>
</input>
<input type='keyboard' bus='ps2'>
<alias name='input1'/>
</input>
<graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</video>
<memballoon model='virtio'>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</memballoon>
<iommu model='intel'>
<driver intremap='on' caching_mode='on' iotlb='on'/>
</iommu>
</devices>
<seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>

4. start testpmd inside guest
dpdk-testpmd -l 0-2 -n 1 --socket-mem 1024 – -i --forward-mode=io --burst=32 --rxd=8192 --txd=8192 --max-pkt-len=9600 --mbuf-size=9728 --nb-cores=2 --rxq=1 --txq=1 --mbcache=512 --auto-start

5.send traffic with T-rex sender
./binary-search.py --traffic-generator=trex-txrx --frame-size=64 --num-flows=1024 --max-loss-pct=0 --search-runtime=10 --validation-runtime=60 --rate-tolerance=10 --runtime-tolerance=10 --rate=25 --rate-unit=% --duplicate-packet-failure=retry-to-fail --negative-packet-loss=retry-to-fail --warmup-trial --warmup-trial-runtime=10 --rate=25 --rate-unit=% --one-shot=0 --use-src-ip-flows=1 --use-dst-ip-flows=1 --use-src-mac-flows=1 --use-dst-mac-flows=1 --send-teaching-measurement --send-teaching-warmup --teaching-warmup-packet-type=generic --teaching-measurement-packet-type=generic --teaching-warmup-packet-rate=1000 --use-src-mac-flows=0 --use-dst-mac-flows=0 --dst-macs=3c:fd:fe:ad:bf:c5,3c:fd:fe:ad:bf:c4 --src-macs=3c:fd:fe:ad:bf:c4,3c:fd:fe:ad:bf:c5

Actual results:
some case got the very low performance.
2q 4pmd viommu 64byte case: 0.428mpps
4q 8pmd viommu 64byte case: 0.665mpps
2q 4pmd viommu 256byte case: 0.89mpps
4q 8pmd viommu 256byte case: 0.78mpps
https://beaker.engineering.redhat.com/jobs/9098999
https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/04/90989/9098999/15847940/175715044/i40e_25.html

Expected results:
All cases got the normal result.