Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-3015

Test Coverage: [NFV] No Performance improvement while scaling the PMD threads in Networker node with OVS-DPDK

    • Icon: Task Task
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • ovs-dpdk
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      ( ) The test coverage is aligned with the epic's acceptance criteria

      Show
      ( ) The test coverage is aligned with the epic's acceptance criteria
    • None

      This task is tracking the test case writing activities to cover the bug described below.

      Reference to BUG: OSPRH-19422, I observed there is no performance througput improvement while scaling the PMD cores from 2 to 14 in each dpdk port of Networker node.

      Reproduced Steps:

      • In the OvsDpdk Networker node, scale PMD threads of geneve port (dpdk2) and external provider (dpdk3) from 2 queue to 14 queue dedicated with isolated CPU alligned to same NUMA node.
      PCI address for dpdk2: 0000:b1:00.0
      NUMA node for PCI device: 1
      Selected CPUs for RX queues on dpdk2: 11 13 15 17 19 21 23 25 27 29 31 33 35 37
      Setting 14 RX queues with PMD affinity: 0:11,1:13,2:15,3:17,4:19,5:21,6:23,7:25,8:27,9:29,10:31,11:33,12:35,13:37
      Configuration applied to dpdk2.
      PCI address for dpdk3: 0000:b1:00.1
      NUMA node for PCI device: 1
      Selected CPUs for RX queues on dpdk3: 39 41 43 45 47 49 51 53 55 57 59 61 63 65
      Setting 14 RX queues with PMD affinity: 0:39,1:41,2:43,3:45,4:47,5:49,6:51,7:53,8:55,9:57,10:59,11:61,12:63,13:65
      Configuration applied to dpdk3.
      

       

      ovs-vsctl get Open_vSwitch . other_config {dpdk-extra=" -n 4", dpdk-init="true", dpdk-socket-limit="4096,4096", dpdk-socket-mem="4096,4096", ovn-chassis-idx-8f41355d-29fb-423c-8706-3516d544cab9="", pmd-cpu-mask=fffffffffffffffc00, vlan-limit="0"}
      
      • In OvsDpdk Compute node keep remain with 2Rxq.
      • Executed the binary-search test with 0.002% loss scenario. Based on the binary-search.txt.xz log file, the TRex-based binary search converged on a maximum rate of approximately 2.109% of line rate (equivalent to ~0.784 MPPS per direction, ~376 Mbps L2, ~527 Mbps L1 per device pair) where packet loss remained below the 0.002% threshold across all trials. This was validated in the final 120-second trial (032) with negligible loss (e.g., 11-13 packets lost across directions).
      binary-search.py -traffic-generator trex-txrx --search-runtime 30 --validation-runtime 120 --max-loss-pct 0.002 --one-shot 0 --rate-unit % --rate 100 --negative-packet-loss retry-to-fail --send-teaching-warmup --disable-upward-search --warmup-trial --warmup-trial-runtime=30 --rate-tolerance-failure fail --duplicate-packet-failure fail --dst-macs fa:16:3e:8b:07:c0,fa:16:3e:c1:72:89,fa:16:3e:96:c9:ba,fa:16:3e:af:15:17 --use-src-mac-flows 0 --use-dst-mac-flows 0 --use-src-ip-flows 0 --use-dst-ip-flows 0 --use-src-port-flows 1 --use-dst-port-flows 1 --num-flows 1024 --frame-size 64 --traffic-direction bidirectional --compress-files --result-output=none --process-all-profiler-data  output-dir /tmp/tmp.2RAUfCbyTn/iteration-1/sample-1  -device-pairs=0:1,2:3 --active-device-pairs=0:1,2:3
      • No explicit errors were found in OVS logs during the test 
      • Environment:
        • OS/Platform: RHOSO-18.0.8 (RHEL-like, kernel 5.14.0-427.13.1.el9_4.x86_64).
        • OVS Version: OVS 3.5.1-5.el9fdp.
        • Topology: TRex Traffic Generator → OVN-DPDK Networker Node → OVN Logical Router (LR) → DUT Compute Node with 2 Grout VMs (handling application-level routing and return traffic). Bi-directional traffic via paired ports (0:1 and 2:3).
        • Benchmark Tool: bench-trafficgen (from https://github.com/perftool-incubator/bench-trafficgen/tree/main/trafficgen), using TRex in software mode off, Mellanox support off.
        • Config Highlights:
          • Packet: 64-byte UDP, 1024 flows (varying src/dst ports).
          • Loss Tolerance: 0.002%.
            Run ID: trafficgen-{}[f0a0e273-1066-4f8a-bf18-fc7df9a0a968|].

      Actual Results: No improvement in PPS rate.

      Attachments: Updating the comment soon

              ovs-qe Openvswtich Quality Engineering Bot
              rhn-support-psahoo Pradipta Sahoo
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: