Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-2556

Test Coverage: ice driver: running ovs dpdk pvp cross numa performance tests takes a long time to get results, and some cases do not get result

    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      ( ) The test coverage is aligned with the epic's acceptance criteria

      Given a RHEL 9.4 system with an ICE 25 GbE NIC and OVS-DPDK configured for the standard cross-NUMA PVP test matrix, 

      When the cross-NUMA PVP performance suite is executed,

      Then, every sub-case finishes without abnormal packet loss and the full run completes in 8 hours or less.

      Show
      ( ) The test coverage is aligned with the epic's acceptance criteria Given a RHEL 9.4 system with an ICE 25 GbE NIC and OVS-DPDK configured for the standard cross-NUMA PVP test matrix,  When the cross-NUMA PVP performance suite is executed, Then, every sub-case finishes without abnormal packet loss and the full run completes in 8 hours or less.
    • rhel-9
    • None
    • rhel-net-ovs-dpdk

      This task is tracking the test case writing activities to cover the bug described below.

       Problem Description: Clearly explain the issue.

      ice driver: running ovs dpdk pvp cross numa performance tests takes a long time to get results, and some cases do not get result
       

       Impact Assessment: Describe the severity and impact (e.g., network down,availability of a workaround, etc.).

      It take long time to run the ovs dpdk pvp cross numa case

       Software Versions: Specify the exact versions in use (e.g.,openvswitch3.1-3.1.0-147.el8fdp).

      rhel9.4
      openvswitch3.3-3.3.0-76.el9fdp
       

        Issue Type: Indicate whether this is a new issue or a regression (if a regression, state the last known working version).

      fdp24.H cross numa performance test: It took 6.5 hours to run all cross numa case and all cases got the normal results.
      https://beaker.engineering.redhat.com/jobs/10248067
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2024/11/102480/10248067/17555757/187388195/ice_25.html

      But fdp25.a, it took more than 25 hours to complete, and the case(ovs-dpdk-vhostuser-pvp-queue4-pmds8-vcpus9-size256-vIOMMU-no-vlan11) does not get result.
      https://beaker.engineering.redhat.com/jobs/10510758
      http://netqe-bj.usersys.redhat.com/share/tli/bash_perf_result/ice_25_job_10510758.html

      It may be a regression, need rerun the performance with fdp24.H version to confirm.
       

       Reproducibility: Confirm if the issue can be reproduced consistently. If not, describe how often it occurs.

      It took long time to complete the jobs, so just run one time. i will run more jobs when the system is available.
       

       Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

      Build ovs-dpdk-vhostuser-pvp topo, and guest use the cpu that cross numa and used trex send rfc2544 traffic to get the throughput and latency performance.
      One job contain following setting.
      1q 2pmd 3vcpus noviommu with vlan tag traffic
      1q 4pmd 3vcpus noviommu with vlan tag traffic
      2q 4pmd 5vcpus noviommu with vlan tag traffic
      4q 8pmd 9vcpus noviommu with vlan tag traffic
      1q 2pmd 3vcpus viommu with novlan tag traffic
      1q 4pmd 3vcpus viommu with novlan tag traffic
      2q 4pmd 5vcpus viommu with novlan tag traffic
      4q 8pmd 9vcpus viommu with novlan tag traffic

       Expected Behavior: Describe what should happen under normal circumstances.

      It should be take about 6-7 hours(simiiar with the fdp24.H) to finish the cross numa performance tests.
       

       Observed Behavior: Explain what actually happens.

      Because there is always packet loss during testing, trex binary-search.py will try many times, which takes a long time.
       

       Troubleshooting Actions: Outline the steps taken to diagnose or resolve the issue so far.

      I have not troubleshoot it, just check the detail test log, there is always packet loss according to the log.
       

       Logs: If you collected logs please provide them (e.g. sos report, /var/log/openvswitch/* , testpmd console)

              ovsdpdk-triage ovsdpdk triage
              nstbot NST Bot
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: