-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
None
-
None
-
False
-
-
False
-
rhel-sst-network-fastdatapath
-
-
-
ssg_networking
Description of problem:
Version-Release numemphasized textber of selected component (if applicable):
rhel9.2
How reproducible:
Run ovs dpdk vhostuser 4queue 8pmd vlan case.
Steps to Reproduce:
1. Build ovs dpdk vhostuser pvp topo
Bridge ovsbr0
datapath_type: netdev
Port ovsbr0
Interface ovsbr0
type: internal
Port dpdk0
Interface dpdk0
type: dpdk
options:
Port vhost1
Interface vhost1
type: dpdkvhostuserclient
options:
Port vhost0
Interface vhost0
type: dpdkvhostuserclient
options:
Port dpdk1
Interface dpdk1
type: dpdk
options:
ovs_version: "3.2.3"
openflows setting:
ovs-ofctl add-flow ovsbr0 "in_port=dpdk0 actions=output:vhost0"
ovs-ofctl add-flow ovsbr0 "in_port=dpdk1 actions=output:vhost1"
ovs-ofctl add-flow ovsbr0 "in_port=vhost0 actions=output:dpdk0"
ovs-ofctl add-flow ovsbr0 "in_port=vhost1 actions=output:dpdk1"
2. start guest with following xml
<domain type='kvm'> <name>g1</name> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB'/> </hugepages> <locked/> <access mode='shared'/> </memoryBacking> <vcpu placement='static'>9</vcpu> <cputune> <vcpupin vcpu='0' cpuset='19'/> <vcpupin vcpu='1' cpuset='17'/> <vcpupin vcpu='2' cpuset='15'/> <vcpupin vcpu='3' cpuset='13'/> <vcpupin vcpu='4' cpuset='11'/> <vcpupin vcpu='5' cpuset='9'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='5'/> <vcpupin vcpu='8' cpuset='3'/> <emulatorpin cpuset='0,28'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='q35'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pmu state='off'/> <vmport state='off'/> <ioapic driver='qemu'/> </features> <cpu mode='host-passthrough' check='none'> <feature policy='require' name='tsc-deadline'/> <numa> <cell id='0' cpus='0-8' memory='8388608' unit='KiB' memAccess='shared'/> </numa> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/g1.qcow2'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='none'> <alias name='usb'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x8'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x9'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xa'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xb'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:01:02:03'/> <source bridge='virbr0'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='00:de:ad:00:00:01'/> <source type='unix' path='/tmp/vhostuser/vhost0' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='4' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'> <host mrg_rxbuf='off'/> </driver> <address type='pci' domain='0x0000' bus='0x3' slot='0x00' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='00:de:ad:00:00:02'/> <source type='unix' path='/tmp/vhostuser/vhost1' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='4' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'> <host mrg_rxbuf='off'/> </driver> <address type='pci' domain='0x0000' bus='0x4' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </memballoon> <iommu model='intel'> <driver intremap='on' caching_mode='on' iotlb='on'/> </iommu> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'/> </domain>
3. start testpmd inside guest
dpdk-testpmd -l 0-8 -n 1 --socket-mem 1024 – -i --forward-mode=io --burst=32 --rxd=8192 --txd=8192 --max-pkt-len=9600 --mbuf-size=9728 --nb-cores=8 --rxq=4 --txq=4 --mbcache=512 --auto-start
4. send traffic on T-rex server
./binary-search.py --traffic-generator=trex-txrx --frame-size=2000 --num-flows=1024 --max-loss-pct=0 --search-runtime=10 --validation-runtime=60 --rate-tolerance=10 --runtime-tolerance=10 --rate=25 --rate-unit=% --duplicate-packet-failure=retry-to-fail --negative-packet-loss=retry-to-fail --warmup-trial --warmup-trial-runtime=10 --rate=25 --rate-unit=% --one-shot=0 --use-src-ip-flows=1 --use-dst-ip-flows=1 --use-src-mac-flows=1 --use-dst-mac-flows=1 --send-teaching-measurement --send-teaching-warmup --teaching-warmup-packet-type=generic --teaching-measurement-packet-type=generic --teaching-warmup-packet-rate=1000
Actual results:
rhel9.2 ovs3.2:
https://beaker.engineering.redhat.com/jobs/9308287
https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93082/9308287/16175560/177938854/ice_25.html
https://beaker.engineering.redhat.com/jobs/9307920
https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93079/9307920/16175070/177934872/ice_25.html
https://beaker.engineering.redhat.com/jobs/9303582
https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93035/9303582/16169266/177903822/ice_25.html
https://beaker.engineering.redhat.com/jobs/9308585
https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93085/9308585/16175941/177941113/ice_25.html
https://beaker.engineering.redhat.com/jobs/9309302
https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93093/9309302/16177254/177950249/ice_25.html
https://beaker.engineering.redhat.com/jobs/9309308
https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93093/9309308/16177261/177950264/ice_25.html
When the case cannot get the result, check the pmd usage log on system:
packets received: 46738872 packet recirculations: 0 avg. datapath passes per packet: 1.00 phwol hits: 0 mfex opt hits: 0 simple match hits: 46738870 emc hits: 0 smc hits: 0 megaflow hits: 0 avg. subtable lookups per megaflow hit: 0.00 miss with success upcall: 2 miss with failed upcall: 0 avg. packets per output batch: 7.18 idle cycles: 57312064862 (63.21%) processing cycles: 33364120746 (36.79%) avg cycles per packet: 1940.06 (90676185608/46738872) avg processing cycles per packet: 713.84 (33364120746/46738872) pmd thread numa_id 0 core_id 18: packets received: 46914705 packet recirculations: 0 avg. datapath passes per packet: 1.00 phwol hits: 0 mfex opt hits: 0 simple match hits: 46914703 emc hits: 0 smc hits: 0 megaflow hits: 0 avg. subtable lookups per megaflow hit: 0.00 miss with success upcall: 2 miss with failed upcall: 0 avg. packets per output batch: 7.06 idle cycles: 57314806234 (63.21%) processing cycles: 33361474778 (36.79%) avg cycles per packet: 1932.79 (90676281012/46914705) avg processing cycles per packet: 711.11 (33361474778/46914705) pmd thread numa_id 0 core_id 20: packets received: 46010836 packet recirculations: 0 avg. datapath passes per packet: 1.00 phwol hits: 0 mfex opt hits: 0 simple match hits: 46010834 emc hits: 0 smc hits: 0 megaflow hits: 0 avg. subtable lookups per megaflow hit: 0.00 miss with success upcall: 2 miss with failed upcall: 0 avg. packets per output batch: 6.82 idle cycles: 58061565296 (64.03%) processing cycles: 32614761282 (35.97%) avg cycles per packet: 1970.76 (90676326578/46010836) avg processing cycles per packet: 708.85 (32614761282/46010836) pmd thread numa_id 0 core_id 22: packets received: 46565230 packet recirculations: 0 avg. datapath passes per packet: 1.00 phwol hits: 0 mfex opt hits: 0 simple match hits: 46565222 emc hits: 0 smc hits: 0 megaflow hits: 3 avg. subtable lookups per megaflow hit: 2.00 miss with success upcall: 5 miss with failed upcall: 0 avg. packets per output batch: 7.20 idle cycles: 57602812626 (63.53%) processing cycles: 33073541946 (36.47%) avg cycles per packet: 1947.30 (90676354572/46565230) avg processing cycles per packet: 710.26 (33073541946/46565230) pmd thread numa_id 0 core_id 40: packets received: 180012 packet recirculations: 0 avg. datapath passes per packet: 1.00 phwol hits: 0 mfex opt hits: 0 simple match hits: 180008 emc hits: 0 smc hits: 0 megaflow hits: 1 avg. subtable lookups per megaflow hit: 2.00 miss with success upcall: 3 miss with failed upcall: 0 avg. packets per output batch: 15.47 idle cycles: 90563307254 (99.88%) processing cycles: 113091746 (0.12%) avg cycles per packet: 503724.19 (90676399000/180012) avg processing cycles per packet: 628.25 (113091746/180012) pmd thread numa_id 0 core_id 42: packets received: 180833 packet recirculations: 0 avg. datapath passes per packet: 1.00 phwol hits: 0 mfex opt hits: 0 simple match hits: 180831 emc hits: 0 smc hits: 0 megaflow hits: 0 avg. subtable lookups per megaflow hit: 0.00 miss with success upcall: 2 miss with failed upcall: 0 avg. packets per output batch: 15.52 idle cycles: 90562949920 (99.87%) processing cycles: 113480830 (0.13%) avg cycles per packet: 501437.41 (90676430750/180833) avg processing cycles per packet: 627.54 (113480830/180833) pmd thread numa_id 0 core_id 44: packets received: 182666 packet recirculations: 0 avg. datapath passes per packet: 1.00 phwol hits: 0 mfex opt hits: 0 simple match hits: 182664 emc hits: 0 smc hits: 0 megaflow hits: 0 avg. subtable lookups per megaflow hit: 0.00 miss with success upcall: 2 miss with failed upcall: 0 avg. packets per output batch: 15.46 idle cycles: 90562766392 (99.87%) processing cycles: 113692496 (0.13%) avg cycles per packet: 496405.78 (90676458888/182666) avg processing cycles per packet: 622.41 (113692496/182666) pmd thread numa_id 0 core_id 46: packets received: 179748 packet recirculations: 0 avg. datapath passes per packet: 1.00 phwol hits: 0 mfex opt hits: 0 simple match hits: 179744 emc hits: 0 smc hits: 0 megaflow hits: 1 avg. subtable lookups per megaflow hit: 2.00 miss with success upcall: 3 miss with failed upcall: 0 avg. packets per output batch: 14.98 idle cycles: 90563311922 (99.88%) processing cycles: 113184984 (0.12%) avg cycles per packet: 504464.57 (90676496906/179748) avg processing cycles per packet: 629.69 (113184984/179748) main thread: packets received: 0 packet recirculations: 0 avg. datapath passes per packet: 0.00 phwol hits: 0 mfex opt hits: 0 simple match hits: 0 emc hits: 0 smc hits: 0 megaflow hits: 0 avg. subtable lookups per megaflow hit: 0.00 miss with success upcall: 0 miss with failed upcall: 0 avg. packets per output batch: 0.00 Displaying last 60 seconds pmd usage % pmd thread numa_id 0 core_id 16: isolated : true port: dpdk0 queue-id: 3 (enabled) pmd usage: 21 % port: vhost0 queue-id: 3 (enabled) pmd usage: 0 % overhead: 2 % pmd thread numa_id 0 core_id 18: isolated : true port: dpdk0 queue-id: 2 (enabled) pmd usage: 20 % port: vhost0 queue-id: 2 (enabled) pmd usage: 0 % overhead: 2 % pmd thread numa_id 0 core_id 20: isolated : true port: dpdk0 queue-id: 1 (enabled) pmd usage: 20 % port: vhost0 queue-id: 1 (enabled) pmd usage: 0 % overhead: 2 % pmd thread numa_id 0 core_id 22: isolated : true port: dpdk0 queue-id: 0 (enabled) pmd usage: 20 % port: vhost0 queue-id: 0 (enabled) pmd usage: 0 % overhead: 2 % pmd thread numa_id 0 core_id 40: isolated : true port: dpdk1 queue-id: 3 (enabled) pmd usage: 0 % port: vhost1 queue-id: 3 (enabled) pmd usage: 0 % overhead: 0 % pmd thread numa_id 0 core_id 42: isolated : true port: dpdk1 queue-id: 2 (enabled) pmd usage: 0 % port: vhost1 queue-id: 2 (enabled) pmd usage: 0 % overhead: 0 % pmd thread numa_id 0 core_id 44: isolated : true port: dpdk1 queue-id: 1 (enabled) pmd usage: 0 % port: vhost1 queue-id: 1 (enabled) pmd usage: 0 % overhead: 0 % pmd thread numa_id 0 core_id 46: isolated : true port: dpdk1 queue-id: 0 (enabled) pmd usage: 0 % port: vhost1 queue-id: 0 (enabled) pmd usage: 0 % overhead: 0 %