Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-2567

QE verification: ice driver: running ovs dpdk pvp cross numa performance tests takes a long time to get results, and some cases do not get result

    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      ( ) The bug has been reproduced and verified by QE members
      ( ) Test coverage has been added to downstream CI
      ( ) For new feature, failed test plans have bugs added as children to the epic
      ( ) The bug is cloned to any relevant release that we support and/or is needed

      Show
      ( ) The bug has been reproduced and verified by QE members ( ) Test coverage has been added to downstream CI ( ) For new feature, failed test plans have bugs added as children to the epic ( ) The bug is cloned to any relevant release that we support and/or is needed
    • rhel-9
    • None
    • rhel-net-ovs-dpdk

      This ticket is tracking the QE verification effort for the solution to the problem described below.

       Problem Description: Clearly explain the issue.

      ice driver: running ovs dpdk pvp cross numa performance tests takes a long time to get results, and some cases do not get result
       

       Impact Assessment: Describe the severity and impact (e.g., network down,availability of a workaround, etc.).

      It take long time to run the ovs dpdk pvp cross numa case

       Software Versions: Specify the exact versions in use (e.g.,openvswitch3.1-3.1.0-147.el8fdp).

      rhel9.4
      openvswitch3.3-3.3.0-76.el9fdp
       

        Issue Type: Indicate whether this is a new issue or a regression (if a regression, state the last known working version).

      fdp24.H cross numa performance test: It took 6.5 hours to run all cross numa case and all cases got the normal results.
      https://beaker.engineering.redhat.com/jobs/10248067
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2024/11/102480/10248067/17555757/187388195/ice_25.html

      But fdp25.a, it took more than 25 hours to complete, and the case(ovs-dpdk-vhostuser-pvp-queue4-pmds8-vcpus9-size256-vIOMMU-no-vlan11) does not get result.
      https://beaker.engineering.redhat.com/jobs/10510758
      http://netqe-bj.usersys.redhat.com/share/tli/bash_perf_result/ice_25_job_10510758.html

      It may be a regression, need rerun the performance with fdp24.H version to confirm.
       

       Reproducibility: Confirm if the issue can be reproduced consistently. If not, describe how often it occurs.

      It took long time to complete the jobs, so just run one time. i will run more jobs when the system is available.
       

       Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

      Build ovs-dpdk-vhostuser-pvp topo, and guest use the cpu that cross numa and used trex send rfc2544 traffic to get the throughput and latency performance.
      One job contain following setting.
      1q 2pmd 3vcpus noviommu with vlan tag traffic
      1q 4pmd 3vcpus noviommu with vlan tag traffic
      2q 4pmd 5vcpus noviommu with vlan tag traffic
      4q 8pmd 9vcpus noviommu with vlan tag traffic
      1q 2pmd 3vcpus viommu with novlan tag traffic
      1q 4pmd 3vcpus viommu with novlan tag traffic
      2q 4pmd 5vcpus viommu with novlan tag traffic
      4q 8pmd 9vcpus viommu with novlan tag traffic

       Expected Behavior: Describe what should happen under normal circumstances.

      It should be take about 6-7 hours(simiiar with the fdp24.H) to finish the cross numa performance tests.
       

       Observed Behavior: Explain what actually happens.

      Because there is always packet loss during testing, trex binary-search.py will try many times, which takes a long time.
       

       Troubleshooting Actions: Outline the steps taken to diagnose or resolve the issue so far.

      I have not troubleshoot it, just check the detail test log, there is always packet loss according to the log.
       

       Logs: If you collected logs please provide them (e.g. sos report, /var/log/openvswitch/* , testpmd console)

              ovsdpdk-triage ovsdpdk triage
              nstbot NST Bot
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: