Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-1585

[Investigation spike] cx6 lx card: some ovs dpdk pvp perf test got low performance

    • Icon: Task Task
    • Resolution: Duplicate
    • Icon: Normal Normal
    • None
    • None
    • openvswitch3.5
    • None
    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      Given the issue described in FDP-1572,

      When an engineer investigates and analyzes the logs,

      Then a comment should be added to this Jira ticket summarizing the root cause of these errors and potential solutions for the problem.

      Definition of Done

      ( ) Spend up to 2 hours investigating and analyzing the logs provided
      ( ) Write a comment in this ticket summarizing the root cause of the issue and suggesting fix or follow-up actions

      Show
      Given the issue described in FDP-1572 , When an engineer investigates and analyzes the logs, Then a comment should be added to this Jira ticket summarizing the root cause of these errors and potential solutions for the problem. Definition of Done ( ) Spend up to 2 hours investigating and analyzing the logs provided ( ) Write a comment in this ticket summarizing the root cause of the issue and suggesting fix or follow-up actions
    • rhel-9
    • None

      This ticket is created automatically to track an investigation spike related to the issue: FDP-1572.

      The goal is to identify the root cause or clarify unknowns and evaluate possible solutions or workarounds for the problem described below.

       Problem Description: Clearly explain the issue.

      cx6 lx card: some ovs dpdk pvp perf test got low performance
       

       Impact Assessment: Describe the severity and impact (e.g., network down,availability of a workaround, etc.).

       

       Software Versions: Specify the exact versions in use (e.g.,openvswitch3.1-3.1.0-147.el8fdp).

      openvswitch3.5-3.5.1-42.el9fdp
       openvswitch3.5-3.5.0-13.el10fdp

        Issue Type: Indicate whether this is a new issue or a regression (if a regression, state the last known working version).

       

       Reproducibility: Confirm if the issue can be reproduced consistently. If not, describe how often it occurs.

       

       Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

      Run ovs dpdk pvp perf test job
      build ovs dpdk bridge
      add two dpdk port to ovs bridge
      add two vhostuser port to ovs bridge
      inside guest, start testpmd forward packet
      On trex side, send rfc2544 traffic.
       

       Expected Behavior: Describe what should happen under normal circumstances.

      All cases got normal performance
       

       Observed Behavior: Explain what actually happens.

      some cases got low performance, such following job of openvswitch3.5-3.5.1-42.el9fdp.
      ovs_dpdk_vhostuser_pvp_queues1_pmds2_vcpus3_vIOMMU_no_vlan1 1500byte got 1.0mpps, it should got about 1.8mpps

      ovs_dpdk_vhostuser_pvp_queues1_pmds4_vcpus3_vIOMMU_no_vlan11 1500byte got 1.0mpps, it should got about 2.8mpps

      ovs_dpdk_vhostuser_pvp_queues4_pmds8_vcpus9_vIOMMU_yes_vlan0 256byte got 5.6mpps, it should got about 16mpps

      ovs_dpdk_vhostuser_pvp_queues1_pmds2_vcpus3_vIOMMU_no_vlan11 2000byte got 0.754mpps, it should got about 1.4mpps

      https://beaker.engineering.redhat.com/jobs/11476613
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/08/114766/11476613/19263898/199015352/mlx5_25.html
       

      For following job of openvswitch3.5-3.5.0-13.el10fdp
      ovs_dpdk_vhostuser_pvp_queues1_pmds4_vcpus3_vIOMMU_yes_vlan0 1500byte got 1.0mpps, it should got about 3mpps

      ovs_dpdk_vhostuser_pvp_queues1_pmds2_vcpus3_vIOMMU_no_vlan11 2000byte got 0.775mpps, it should got about 1.4mpps

      https://beaker.engineering.redhat.com/jobs/10997971
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/04/109979/10997971/18600649/194311958/mlx5_25.html

      For ovs3.4 and ovs3.3, it has no issue.
      ovs3.3:
      https://beaker.engineering.redhat.com/jobs/10988728
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/04/109887/10988728/18585889/194183083/mlx5_25.html
      ovs3.4
      https://beaker.engineering.redhat.com/jobs/10993817
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/04/109938/10993817/18594096/194253692/mlx5_25.html

      There is no this issue for cx6 dx card.

       Troubleshooting Actions: Outline the steps taken to diagnose or resolve the issue so far.

       

       Logs: If you collected logs please provide them (e.g. sos report, /var/log/openvswitch/* , testpmd console)

      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/08/114766/11476613/19263897/199015350/ovs-vswitchd.log

              ovsdpdk-triage ovsdpdk triage
              nstbot NST Bot
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: