Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-2543

Upstream: ice driver: running ovs dpdk pvp cross numa performance tests takes a long time to get results, and some cases do not get result

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • None
    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      Please mark each item below with ( / ) if completed or ( x ) if incomplete:
      ( ) Unit test or Integration test case are written and pass successfully


      ( ) The upstream pull request is merged upstream and pass CI

      Show
      Please mark each item below with ( / ) if completed or ( x ) if incomplete: ( ) Unit test or Integration test case are written and pass successfully ( ) The upstream pull request is merged upstream and pass CI
    • rhel-9
    • None
    • rhel-net-ovs-dpdk

      This is tracking the upstream effort needed to deliver the solution to the bug described below.


       Problem Description: Clearly explain the issue.

      ice driver: running ovs dpdk pvp cross numa performance tests takes a long time to get results, and some cases do not get result
       

       Impact Assessment: Describe the severity and impact (e.g., network down,availability of a workaround, etc.).

      It take long time to run the ovs dpdk pvp cross numa case

       Software Versions: Specify the exact versions in use (e.g.,openvswitch3.1-3.1.0-147.el8fdp).

      rhel9.4
      openvswitch3.3-3.3.0-76.el9fdp
       

        Issue Type: Indicate whether this is a new issue or a regression (if a regression, state the last known working version).

      fdp24.H cross numa performance test: It took 6.5 hours to run all cross numa case and all cases got the normal results.
      https://beaker.engineering.redhat.com/jobs/10248067
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2024/11/102480/10248067/17555757/187388195/ice_25.html

      But fdp25.a, it took more than 25 hours to complete, and the case(ovs-dpdk-vhostuser-pvp-queue4-pmds8-vcpus9-size256-vIOMMU-no-vlan11) does not get result.
      https://beaker.engineering.redhat.com/jobs/10510758
      http://netqe-bj.usersys.redhat.com/share/tli/bash_perf_result/ice_25_job_10510758.html

      It may be a regression, need rerun the performance with fdp24.H version to confirm.
       

       Reproducibility: Confirm if the issue can be reproduced consistently. If not, describe how often it occurs.

      It took long time to complete the jobs, so just run one time. i will run more jobs when the system is available.
       

       Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

      Build ovs-dpdk-vhostuser-pvp topo, and guest use the cpu that cross numa and used trex send rfc2544 traffic to get the throughput and latency performance.
      One job contain following setting.
      1q 2pmd 3vcpus noviommu with vlan tag traffic
      1q 4pmd 3vcpus noviommu with vlan tag traffic
      2q 4pmd 5vcpus noviommu with vlan tag traffic
      4q 8pmd 9vcpus noviommu with vlan tag traffic
      1q 2pmd 3vcpus viommu with novlan tag traffic
      1q 4pmd 3vcpus viommu with novlan tag traffic
      2q 4pmd 5vcpus viommu with novlan tag traffic
      4q 8pmd 9vcpus viommu with novlan tag traffic

       Expected Behavior: Describe what should happen under normal circumstances.

      It should be take about 6-7 hours(simiiar with the fdp24.H) to finish the cross numa performance tests.
       

       Observed Behavior: Explain what actually happens.

      Because there is always packet loss during testing, trex binary-search.py will try many times, which takes a long time.
       

       Troubleshooting Actions: Outline the steps taken to diagnose or resolve the issue so far.

      I have not troubleshoot it, just check the detail test log, there is always packet loss according to the log.
       

       Logs: If you collected logs please provide them (e.g. sos report, /var/log/openvswitch/* , testpmd console)


              ovsdpdk-triage ovsdpdk triage
              nstbot NST Bot
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: