Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-629

ice driver: some ovs dpdk pvp case cannot get performance result sometimes

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • rhel-sst-network-fastdatapath
    • ssg_networking

      Description of problem:

      Version-Release numemphasized textber of selected component (if applicable):
      rhel9.2

      How reproducible:

      Run ovs dpdk vhostuser 4queue 8pmd vlan case.
      Steps to Reproduce:
      1. Build ovs dpdk vhostuser pvp topo
      Bridge ovsbr0
      datapath_type: netdev
      Port ovsbr0
      Interface ovsbr0
      type: internal
      Port dpdk0
      Interface dpdk0
      type: dpdk
      options:

      {dpdk-devargs="0000:07:00.0", n_rxq="4", n_rxq_desc="1024", n_txq_desc="1024"}

      Port vhost1
      Interface vhost1
      type: dpdkvhostuserclient
      options:

      {vhost-server-path="/tmp/vhostuser/vhost1"}

      Port vhost0
      Interface vhost0
      type: dpdkvhostuserclient
      options:

      {vhost-server-path="/tmp/vhostuser/vhost0"}

      Port dpdk1
      Interface dpdk1
      type: dpdk
      options:

      {dpdk-devargs="0000:07:00.1", n_rxq="4", n_rxq_desc="1024", n_txq_desc="1024"}

      ovs_version: "3.2.3"

      openflows setting:
      ovs-ofctl add-flow ovsbr0 "in_port=dpdk0 actions=output:vhost0"
      ovs-ofctl add-flow ovsbr0 "in_port=dpdk1 actions=output:vhost1"
      ovs-ofctl add-flow ovsbr0 "in_port=vhost0 actions=output:dpdk0"
      ovs-ofctl add-flow ovsbr0 "in_port=vhost1 actions=output:dpdk1"

      2. start guest with following xml

      <domain type='kvm'>
        <name>g1</name>
        <memory unit='KiB'>8388608</memory>
        <currentMemory unit='KiB'>8388608</currentMemory>
        <memoryBacking>
          <hugepages>
            <page size='1048576' unit='KiB'/>
          </hugepages>
          <locked/>
          <access mode='shared'/>
        </memoryBacking>
        <vcpu placement='static'>9</vcpu>
        <cputune>
          <vcpupin vcpu='0' cpuset='19'/>
          <vcpupin vcpu='1' cpuset='17'/>
          <vcpupin vcpu='2' cpuset='15'/>
          <vcpupin vcpu='3' cpuset='13'/>
          <vcpupin vcpu='4' cpuset='11'/>
          <vcpupin vcpu='5' cpuset='9'/>
          <vcpupin vcpu='6' cpuset='7'/>
          <vcpupin vcpu='7' cpuset='5'/>
          <vcpupin vcpu='8' cpuset='3'/>
          <emulatorpin cpuset='0,28'/>
        </cputune>
        <numatune>
          <memory mode='strict' nodeset='0'/>
        </numatune>
        <resource>
          <partition>/machine</partition>
        </resource>
        <os>
          <type arch='x86_64' machine='q35'>hvm</type>
          <boot dev='hd'/>
        </os>
        <features>
          <acpi/>
          <apic/>
          <pmu state='off'/>
          <vmport state='off'/>
          <ioapic driver='qemu'/>
        </features>
        <cpu mode='host-passthrough' check='none'>
          <feature policy='require' name='tsc-deadline'/>
          <numa>
            <cell id='0' cpus='0-8' memory='8388608' unit='KiB' memAccess='shared'/>
          </numa>
        </cpu>
        <clock offset='utc'>
          <timer name='rtc' tickpolicy='catchup'/>
          <timer name='pit' tickpolicy='delay'/>
          <timer name='hpet' present='no'/>
        </clock>
        <on_poweroff>destroy</on_poweroff>
        <on_reboot>restart</on_reboot>
        <on_crash>restart</on_crash>
        <pm>
          <suspend-to-mem enabled='no'/>
          <suspend-to-disk enabled='no'/>
        </pm>
        <devices>
          <emulator>/usr/libexec/qemu-kvm</emulator>
          <disk type='file' device='disk'>
            <driver name='qemu' type='qcow2'/>
            <source file='/var/lib/libvirt/images/g1.qcow2'/>
            <backingStore/>
            <target dev='vda' bus='virtio'/>
            <alias name='virtio-disk0'/>
            <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
          </disk>
          <controller type='usb' index='0' model='none'>
            <alias name='usb'/>
          </controller>
          <controller type='pci' index='0' model='pcie-root'>
            <alias name='pcie.0'/>
          </controller>
          <controller type='pci' index='1' model='pcie-root-port'>
            <model name='pcie-root-port'/>
            <target chassis='1' port='0x10'/>
            <alias name='pci.1'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
          </controller>
          <controller type='pci' index='2' model='pcie-root-port'>
            <model name='pcie-root-port'/>
            <target chassis='2' port='0x11'/>
            <alias name='pci.2'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
          </controller>
          <controller type='pci' index='3' model='pcie-root-port'>
            <model name='pcie-root-port'/>
            <target chassis='3' port='0x8'/>
            <alias name='pci.3'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
          </controller>
          <controller type='pci' index='4' model='pcie-root-port'>
            <model name='pcie-root-port'/>
            <target chassis='4' port='0x9'/>
            <alias name='pci.4'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
          </controller>
          <controller type='pci' index='5' model='pcie-root-port'>
            <model name='pcie-root-port'/>
            <target chassis='5' port='0xa'/>
            <alias name='pci.5'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
          </controller>
          <controller type='pci' index='6' model='pcie-root-port'>
            <model name='pcie-root-port'/>
            <target chassis='6' port='0xb'/>
            <alias name='pci.6'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
          </controller>
          <controller type='sata' index='0'>
            <alias name='ide'/>
            <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
          </controller>
          <interface type='bridge'>
            <mac address='52:54:00:01:02:03'/>
            <source bridge='virbr0'/>
            <model type='virtio'/>
          </interface>
          <interface type='vhostuser'>
            <mac address='00:de:ad:00:00:01'/>
            <source type='unix' path='/tmp/vhostuser/vhost0' mode='server'/>
            <model type='virtio'/>
            <driver name='vhost' queues='4' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'>
              <host mrg_rxbuf='off'/>
            </driver>
            <address type='pci' domain='0x0000' bus='0x3' slot='0x00' function='0x0'/>
          </interface>
          <interface type='vhostuser'>
            <mac address='00:de:ad:00:00:02'/>
            <source type='unix' path='/tmp/vhostuser/vhost1' mode='server'/>
            <model type='virtio'/>
            <driver name='vhost' queues='4' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'>
              <host mrg_rxbuf='off'/>
            </driver>
            <address type='pci' domain='0x0000' bus='0x4' slot='0x00' function='0x0'/>
          </interface>
          <serial type='pty'>
            <source path='/dev/pts/1'/>
            <target type='isa-serial' port='0'>
              <model name='isa-serial'/>
            </target>
            <alias name='serial0'/>
          </serial>
          <console type='pty' tty='/dev/pts/1'>
            <source path='/dev/pts/1'/>
            <target type='serial' port='0'/>
            <alias name='serial0'/>
          </console>
          <input type='mouse' bus='ps2'>
            <alias name='input0'/>
          </input>
          <input type='keyboard' bus='ps2'>
            <alias name='input1'/>
          </input>
          <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
            <listen type='address' address='0.0.0.0'/>
          </graphics>
          <video>
            <model type='cirrus' vram='16384' heads='1' primary='yes'/>
            <alias name='video0'/>
            <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
          </video>
          <memballoon model='virtio'>
            <alias name='balloon0'/>
            <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
          </memballoon>
          <iommu model='intel'>
            <driver intremap='on' caching_mode='on' iotlb='on'/>
          </iommu>
        </devices>
        <seclabel type='dynamic' model='selinux' relabel='yes'/>
      </domain>
      

      3. start testpmd inside guest
      dpdk-testpmd -l 0-8 -n 1 --socket-mem 1024 – -i --forward-mode=io --burst=32 --rxd=8192 --txd=8192 --max-pkt-len=9600 --mbuf-size=9728 --nb-cores=8 --rxq=4 --txq=4 --mbcache=512 --auto-start

      4. send traffic on T-rex server
      ./binary-search.py --traffic-generator=trex-txrx --frame-size=2000 --num-flows=1024 --max-loss-pct=0 --search-runtime=10 --validation-runtime=60 --rate-tolerance=10 --runtime-tolerance=10 --rate=25 --rate-unit=% --duplicate-packet-failure=retry-to-fail --negative-packet-loss=retry-to-fail --warmup-trial --warmup-trial-runtime=10 --rate=25 --rate-unit=% --one-shot=0 --use-src-ip-flows=1 --use-dst-ip-flows=1 --use-src-mac-flows=1 --use-dst-mac-flows=1 --send-teaching-measurement --send-teaching-warmup --teaching-warmup-packet-type=generic --teaching-measurement-packet-type=generic --teaching-warmup-packet-rate=1000

      Actual results:
      rhel9.2 ovs3.2:
      https://beaker.engineering.redhat.com/jobs/9308287
      https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93082/9308287/16175560/177938854/ice_25.html
      https://beaker.engineering.redhat.com/jobs/9307920
      https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93079/9307920/16175070/177934872/ice_25.html
      https://beaker.engineering.redhat.com/jobs/9303582
      https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93035/9303582/16169266/177903822/ice_25.html
      https://beaker.engineering.redhat.com/jobs/9308585
      https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93085/9308585/16175941/177941113/ice_25.html
      https://beaker.engineering.redhat.com/jobs/9309302
      https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93093/9309302/16177254/177950249/ice_25.html
      https://beaker.engineering.redhat.com/jobs/9309308
      https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2024/05/93093/9309308/16177261/177950264/ice_25.html
      When the case cannot get the result, check the pmd usage log on system:

        packets received: 46738872
        packet recirculations: 0
        avg. datapath passes per packet: 1.00
        phwol hits: 0
        mfex opt hits: 0
        simple match hits: 46738870
        emc hits: 0
        smc hits: 0
        megaflow hits: 0
        avg. subtable lookups per megaflow hit: 0.00
        miss with success upcall: 2
        miss with failed upcall: 0
        avg. packets per output batch: 7.18
        idle cycles: 57312064862 (63.21%)
        processing cycles: 33364120746 (36.79%)
        avg cycles per packet: 1940.06 (90676185608/46738872)
        avg processing cycles per packet: 713.84 (33364120746/46738872)
      pmd thread numa_id 0 core_id 18:
        packets received: 46914705
        packet recirculations: 0
        avg. datapath passes per packet: 1.00
        phwol hits: 0
        mfex opt hits: 0
        simple match hits: 46914703
        emc hits: 0
        smc hits: 0
        megaflow hits: 0
        avg. subtable lookups per megaflow hit: 0.00
        miss with success upcall: 2
        miss with failed upcall: 0
        avg. packets per output batch: 7.06
        idle cycles: 57314806234 (63.21%)
        processing cycles: 33361474778 (36.79%)
        avg cycles per packet: 1932.79 (90676281012/46914705)
        avg processing cycles per packet: 711.11 (33361474778/46914705)
      pmd thread numa_id 0 core_id 20:
        packets received: 46010836
        packet recirculations: 0
        avg. datapath passes per packet: 1.00
        phwol hits: 0
        mfex opt hits: 0
        simple match hits: 46010834
        emc hits: 0
        smc hits: 0
        megaflow hits: 0
        avg. subtable lookups per megaflow hit: 0.00
        miss with success upcall: 2
        miss with failed upcall: 0
        avg. packets per output batch: 6.82
        idle cycles: 58061565296 (64.03%)
        processing cycles: 32614761282 (35.97%)
        avg cycles per packet: 1970.76 (90676326578/46010836)
        avg processing cycles per packet: 708.85 (32614761282/46010836)
      pmd thread numa_id 0 core_id 22:
        packets received: 46565230
        packet recirculations: 0
        avg. datapath passes per packet: 1.00
        phwol hits: 0
        mfex opt hits: 0
        simple match hits: 46565222
        emc hits: 0
        smc hits: 0
        megaflow hits: 3
        avg. subtable lookups per megaflow hit: 2.00
        miss with success upcall: 5
        miss with failed upcall: 0
        avg. packets per output batch: 7.20
        idle cycles: 57602812626 (63.53%)
        processing cycles: 33073541946 (36.47%)
        avg cycles per packet: 1947.30 (90676354572/46565230)
        avg processing cycles per packet: 710.26 (33073541946/46565230)
      pmd thread numa_id 0 core_id 40:
        packets received: 180012
        packet recirculations: 0
        avg. datapath passes per packet: 1.00
        phwol hits: 0
        mfex opt hits: 0
        simple match hits: 180008
        emc hits: 0
        smc hits: 0
        megaflow hits: 1
        avg. subtable lookups per megaflow hit: 2.00
        miss with success upcall: 3
        miss with failed upcall: 0
        avg. packets per output batch: 15.47
        idle cycles: 90563307254 (99.88%)
        processing cycles: 113091746 (0.12%)
        avg cycles per packet: 503724.19 (90676399000/180012)
        avg processing cycles per packet: 628.25 (113091746/180012)
      pmd thread numa_id 0 core_id 42:
        packets received: 180833
        packet recirculations: 0
        avg. datapath passes per packet: 1.00
        phwol hits: 0
        mfex opt hits: 0
        simple match hits: 180831
        emc hits: 0
        smc hits: 0
        megaflow hits: 0
        avg. subtable lookups per megaflow hit: 0.00
        miss with success upcall: 2
        miss with failed upcall: 0
        avg. packets per output batch: 15.52
        idle cycles: 90562949920 (99.87%)
        processing cycles: 113480830 (0.13%)
        avg cycles per packet: 501437.41 (90676430750/180833)
        avg processing cycles per packet: 627.54 (113480830/180833)
      pmd thread numa_id 0 core_id 44:
        packets received: 182666
        packet recirculations: 0
        avg. datapath passes per packet: 1.00
        phwol hits: 0
        mfex opt hits: 0
        simple match hits: 182664
        emc hits: 0
        smc hits: 0
        megaflow hits: 0
        avg. subtable lookups per megaflow hit: 0.00
        miss with success upcall: 2
        miss with failed upcall: 0
        avg. packets per output batch: 15.46
        idle cycles: 90562766392 (99.87%)
        processing cycles: 113692496 (0.13%)
        avg cycles per packet: 496405.78 (90676458888/182666)
        avg processing cycles per packet: 622.41 (113692496/182666)
      pmd thread numa_id 0 core_id 46:
        packets received: 179748
        packet recirculations: 0
        avg. datapath passes per packet: 1.00
        phwol hits: 0
        mfex opt hits: 0
        simple match hits: 179744
        emc hits: 0
        smc hits: 0
        megaflow hits: 1
        avg. subtable lookups per megaflow hit: 2.00
        miss with success upcall: 3
        miss with failed upcall: 0
        avg. packets per output batch: 14.98
        idle cycles: 90563311922 (99.88%)
        processing cycles: 113184984 (0.12%)
        avg cycles per packet: 504464.57 (90676496906/179748)
        avg processing cycles per packet: 629.69 (113184984/179748)
      main thread:
        packets received: 0
        packet recirculations: 0
        avg. datapath passes per packet: 0.00
        phwol hits: 0
        mfex opt hits: 0
        simple match hits: 0
        emc hits: 0
        smc hits: 0
        megaflow hits: 0
        avg. subtable lookups per megaflow hit: 0.00
        miss with success upcall: 0
        miss with failed upcall: 0
        avg. packets per output batch: 0.00
      Displaying last 60 seconds pmd usage %
      pmd thread numa_id 0 core_id 16:
        isolated : true
        port: dpdk0             queue-id:  3 (enabled)   pmd usage: 21 %
        port: vhost0            queue-id:  3 (enabled)   pmd usage:  0 %
        overhead:  2 %
      pmd thread numa_id 0 core_id 18:
        isolated : true
        port: dpdk0             queue-id:  2 (enabled)   pmd usage: 20 %
        port: vhost0            queue-id:  2 (enabled)   pmd usage:  0 %
        overhead:  2 %
      pmd thread numa_id 0 core_id 20:
        isolated : true
        port: dpdk0             queue-id:  1 (enabled)   pmd usage: 20 %
        port: vhost0            queue-id:  1 (enabled)   pmd usage:  0 %
        overhead:  2 %
      pmd thread numa_id 0 core_id 22:
        isolated : true
        port: dpdk0             queue-id:  0 (enabled)   pmd usage: 20 %
        port: vhost0            queue-id:  0 (enabled)   pmd usage:  0 %
        overhead:  2 %
      pmd thread numa_id 0 core_id 40:
        isolated : true
        port: dpdk1             queue-id:  3 (enabled)   pmd usage:  0 %
        port: vhost1            queue-id:  3 (enabled)   pmd usage:  0 %
        overhead:  0 %
      pmd thread numa_id 0 core_id 42:
        isolated : true
        port: dpdk1             queue-id:  2 (enabled)   pmd usage:  0 %
        port: vhost1            queue-id:  2 (enabled)   pmd usage:  0 %
        overhead:  0 %
      pmd thread numa_id 0 core_id 44:
        isolated : true
        port: dpdk1             queue-id:  1 (enabled)   pmd usage:  0 %
        port: vhost1            queue-id:  1 (enabled)   pmd usage:  0 %
        overhead:  0 %
      pmd thread numa_id 0 core_id 46:
        isolated : true
        port: dpdk1             queue-id:  0 (enabled)   pmd usage:  0 %
        port: vhost1            queue-id:  0 (enabled)   pmd usage:  0 %
        overhead:  0 %
      

              rhn-support-dmarchan David Marchand
              tli@redhat.com Ting Li
              Ting Li Ting Li
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: