Uploaded image for project: 'Fast Datapath Product'
  1. Fast Datapath Product
  2. FDP-2545

Test Coverage: cx6 lx card: some ovs dpdk pvp performance has no result when running gating CI testing on rhel10

    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      ( ) The test coverage is aligned with the epic's acceptance criteria

      Show
      ( ) The test coverage is aligned with the epic's acceptance criteria
    • rhel-10
    • None
    • rhel-net-ovs-dpdk

      This task is tracking the test case writing activities to cover the bug described below.

       Problem Description: Clearly explain the issue.

      When running gating CI on cx6 LX card, some cases has no performance result on rhel10
       

       Impact Assessment: Describe the severity and impact (e.g., network down,availability of a workaround, etc.).

       

       Software Versions: Specify the exact versions in use (e.g.,openvswitch3.1-3.1.0-147.el8fdp).

      os: rhel10.0 and rhel10.1
       openvswitch3.6-3.6.0-9.el10fdp.x86_64
      openvswitch3.5-3.5.2-45.el10fdp.x86_64

        Issue Type: Indicate whether this is a new issue or a regression (if a regression, state the last known working version).

      It is not a new issue, it also exist on previous ovs build
       

       Reproducibility: Confirm if the issue can be reproduced consistently. If not, describe how often it occurs.

      Most of them are reproducible, It occurs with both CX6 and i40e when using TRex, so it should be unrelated to TRex.
       

       Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

      Run gating CI testing(ovs dpdk pvp performance testing, 1q2pmd, 2q 4pmd, 4q 8pmd cases)
       

       Expected Behavior: Describe what should happen under normal circumstances.

      All cases have performance results.
       

       Observed Behavior: Explain what actually happens.

      Some cases has no result.

      Following jobs trex use i40e card, and some cases has no result.
      DUT cx6 LX: rhel10.0
      Trex i40e: rhel10.0 v2.82
      ovs_dpdk_vhostuser_pvp_queues2_pmds4_vcpus5_vIOMMU_yes_vlan0 and ovs_dpdk_vhostuser_pvp_queues1_pmds2_vcpus3_vIOMMU_no_vlan11 has no result.
      https://beaker.engineering.redhat.com/jobs/11869968
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/11/118699/11869968/19906720/205115603/mlx5_25.html

      DUT cx6 LX: rhel10.0
      Trex i40e: rhel10.0 v2.82
      https://beaker.engineering.redhat.com/jobs/11869399
      https://beaker.engineering.redhat.com/jobs/11863792
       
      DUT cx6 LX: rhel10.0
      Trex i40e: rhel10.0 v3.03
      https://beaker.engineering.redhat.com/jobs/11858069

      DUT cx6 LX: rhel10.0
      Trex i40e: rhel9.4 v2.82
      https://beaker.engineering.redhat.com/jobs/11869974

      Following jobs trex use cx6 card, some cases has no results.
      DUT cx6 LX: rhel10.0
      Trex cx6 DX: rhel9.4 v3.03
      https://beaker.engineering.redhat.com/jobs/11856721
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118567/11856721/19883429/204899854/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11853860
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118538/11853860/19878301/204848887/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11853859
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118538/11853859/19878299/204848883/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11853857
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118538/11853857/19878295/204848856/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11871285

      DUT cx6 LX: rhel10.0
      Trex cx6 DX: rhel10.0 v3.03
      https://beaker.engineering.redhat.com/jobs/11852384
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118523/11852384/19875930/204829293/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11852354
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118523/11852354/19875881/204829008/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11836418
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118364/11836418/19850639/204599504/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11852384
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118523/11852384/19875930/204829293/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11840682
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118406/11840682/19857136/204657625/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11852354
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/10/118523/11852354/19875881/204829008/mlx5_25.html

      DUT cx6 LX: rhel10.0
      Trex cx6 DX: rhel9.7 v3.03
      https://beaker.engineering.redhat.com/jobs/11605585

      rhel10.1 also has issue:
      DUT cx6 LX: rhel10.1
      Trex cx6 DX: rhel9.7 v3.03
      https://beaker.engineering.redhat.com/jobs/11651732
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/09/116517/11651732/19551253/201757690/mlx5_25.html

      DUT cx6 LX: rhel10.1
      Trex cx6 DX: rhel10.1 v3.03
      https://beaker.engineering.redhat.com/jobs/11643750
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/09/116437/11643750/19537216/201591581/mlx5_25.html

      There is no this issue on rhel9.
      rhel9.4:
      DUT cx6 LX: rhel9.4
      Trex cx6 DX: rhel9.4 v2.82
      https://beaker.engineering.redhat.com/jobs/11869972
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/11/118699/11869972/19906726/205115619/mlx5_25.html

      DUT cx6 LX: rhel9.4
      Trex cx6 DX: rhel9.4 v3.03
      https://beaker.engineering.redhat.com/jobs/11869970
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/11/118699/11869970/19906723/205115611/mlx5_25.html

      https://beaker.engineering.redhat.com/jobs/11856145

      rhel9.6:
      https://beaker.engineering.redhat.com/jobs/11627446

      rhel9.7:
      DUT cx6 LX: rhel9.7
      Trex cx6 DX: rhel9.7 v3.03
      https://beaker.engineering.redhat.com/jobs/11664841
      https://beaker.engineering.redhat.com/jobs/11620457

       Troubleshooting Actions: Outline the steps taken to diagnose or resolve the issue so far.

      For example, https://beaker.engineering.redhat.com/jobs/11869968,
       https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/11/118699/11869968/19906720/205115603/mlx5_25.html
      ovs_dpdk_vhostuser_pvp_queues2_pmds4_vcpus5_vIOMMU_yes_vlan0 and ovs_dpdk_vhostuser_pvp_queues1_pmds2_vcpus3_vIOMMU_no_vlan11 has no result, and check DUT the pmd usage log, the pmd usage is 0.
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/11/118699/11869968/19906719/205115600/2_4_5_128_0_pmd.log
      check the Trex log, the send trex traffic failed at the time.

      python ./binary-search.py --traffic-generator=trex-txrx --frame-size=64 --num-flows=1024 --max-loss-pct=0 --search-runtime=10 --validation-runtime=60 --rate-tolerance=10 --runtime-tolerance=10 --rate=25 --rate-unit=% --duplicate-packet-failure=retry-to-fail --negative-packet-loss=retry-to-fail --warmup-trial --warmup-trial-runtime=10 --rate=25 --rate-unit=% --one-shot=0 --use-src-ip-flows=1 --use-dst-ip-flows=1 --use-src-mac-flows=1 --use-dst-mac-flows=1 --send-teaching-measurement --send-teaching-warmup --teaching-warmup-packet-type=generic --teaching-warmup-packet-rate=10000 --use-src-ip-flows=1 --use-dst-ip-flows=1 --use-src-mac-flows=1 --use-dst-mac-flows=0 --use-device-stats
      /var/opt/trex/current/external_libs/scapy-2.3.1/python3/scapy/fields.py:205: SyntaxWarning: "is" with a literal. Did you mean "=="?
        if type(x) is str and len(x) is 6:
      /var/opt/trex/current/external_libs/scapy-2.3.1/python3/scapy/fields.py:207: SyntaxWarning: "is" with a literal. Did you mean "=="?
        if type(x) is bytes and len(x) is 6:
      [2025-11-02 10:02:04.929696][BSO] Namespace(output_dir='./', frame_size='64', num_flows=1024, use_src_ip_flows=1, use_dst_ip_flows=1, use_src_mac_flows=1, use_dst_mac_flows=0, use_src_port_flows=0, use_dst_port_flows=0, use_protocol_flows=0, use_encap_src_ip_flows=0, use_encap_dst_ip_flows=0, use_encap_src_mac_flows=0, use_encap_dst_mac_flows=0, one_shot=0, traffic_direction='bidirectional', validation_runtime=60, search_runtime=10, sniff_runtime=0, rate=25.0, min_rate=0.0, rate_unit='%', packet_protocol='UDP', rate_tolerance=10.0, duplicate_packet_failure_mode='retry-to-fail', rate_tolerance_failure='quit', runtime_tolerance=10.0, negative_packet_loss_mode='retry-to-fail', search_granularity=0.1, max_loss_pct=0.0, src_ports='', dst_ports='', dst_macs='', src_macs='', encap_dst_macs='', encap_src_macs='', dst_ips='', src_ips='', vxlan_ids='', vlan_ids='', encap_dst_ips='', encap_src_ips='', traffic_generator='trex-txrx', measure_latency=1, latency_rate=1000, trial_gap=0, max_retries=1, loss_granularity='direction', stream_mode='continuous', use_device_stats=True, enable_segment_monitor=False, device_pairs='0:1', active_device_pairs='0:1', latency_device_pair='--', enable_flow_cache=True, send_teaching_warmup=True, send_teaching_measurement=True, teaching_measurement_interval=10.0, teaching_warmup_packet_rate=10000, teaching_measurement_packet_rate=1000, teaching_warmup_packet_type='generic', teaching_measurement_packet_type='', traffic_profile='', random_seed=0.8013318922453859, pre_trial_cmd='', enable_trex_profiler=True, trex_profiler_interval=3.0, process_all_profiler_data=False, repeat_final_validation=False, warmup_trial=True, warmup_trial_runtime=10, warmup_traffic_profile='', disable_upward_search=False, trex_host='localhost', no_promisc=False, result_output='device', compress_files=False)
      [2025-11-02 10:02:04.930065][BSO] output_dir=./
      [2025-11-02 10:02:04.930180][BSO] CONFIG | traffic_generator=trex-txrx
      [2025-11-02 10:02:04.930184][BSO] CONFIG | rate=25.0
      [2025-11-02 10:02:04.930186][BSO] CONFIG | runtime_tolerance=10.0
      [2025-11-02 10:02:04.930187][BSO] CONFIG | rate_tolerance_failure=quit
      [2025-11-02 10:02:04.930188][BSO] CONFIG | rate_tolerance=10.0
      [2025-11-02 10:02:04.930189][BSO] CONFIG | negative_packet_loss_mode=retry-to-fail
      [2025-11-02 10:02:04.930190][BSO] CONFIG | duplicate_packet_failure_mode=retry-to-fail
      [2025-11-02 10:02:04.930191][BSO] CONFIG | one_shot=0
      [2025-11-02 10:02:04.930193][BSO] CONFIG | min_rate=0.0
      [2025-11-02 10:02:04.930195][BSO] CONFIG | max_loss_pct=0.0
      [2025-11-02 10:02:04.930196][BSO] CONFIG | trial_gap=0
      [2025-11-02 10:02:04.930197][BSO] CONFIG | search_runtime=10
      [2025-11-02 10:02:04.930198][BSO] CONFIG | validation_runtime=60
      [2025-11-02 10:02:04.930199][BSO] CONFIG | sniff_runtime=0
      [2025-11-02 10:02:04.930201][BSO] CONFIG | search_granularity=0.1
      [2025-11-02 10:02:04.930202][BSO] CONFIG | max_retries=1
      [2025-11-02 10:02:04.930203][BSO] CONFIG | loss_granularity=direction
      [2025-11-02 10:02:04.930204][BSO] CONFIG | pre_trial_cmd=
      [2025-11-02 10:02:04.930206][BSO] CONFIG | repeat_final_validation=False
      [2025-11-02 10:02:04.930207][BSO] CONFIG | warmup_trial=True
      [2025-11-02 10:02:04.930208][BSO] CONFIG | warmup_trial_runtime=10
      [2025-11-02 10:02:04.930209][BSO] CONFIG | disable_upward_search=False
      [2025-11-02 10:02:04.930211][BSO] CONFIG | result_output=device
      [2025-11-02 10:02:04.930212][BSO] CONFIG | compress_files=False
      [2025-11-02 10:02:04.930217][BSO] CONFIG | traffic_direction=bidirectional
      [2025-11-02 10:02:04.930218][BSO] CONFIG | latency_traffic_direction=bidirectional
      [2025-11-02 10:02:04.930219][BSO] CONFIG | num_flows=1024
      [2025-11-02 10:02:04.930220][BSO] CONFIG | frame_size=64
      [2025-11-02 10:02:04.930221][BSO] CONFIG | use_src_mac_flows=1
      [2025-11-02 10:02:04.930222][BSO] CONFIG | use_dst_mac_flows=0
      [2025-11-02 10:02:04.930223][BSO] CONFIG | use_src_ip_flows=1
      [2025-11-02 10:02:04.930225][BSO] CONFIG | use_dst_ip_flows=1
      [2025-11-02 10:02:04.930226][BSO] CONFIG | use_src_port_flows=0
      [2025-11-02 10:02:04.930229][BSO] CONFIG | use_dst_port_flows=0
      [2025-11-02 10:02:04.930230][BSO] CONFIG | src_macs=
      [2025-11-02 10:02:04.930231][BSO] CONFIG | dst_macs=
      [2025-11-02 10:02:04.930232][BSO] CONFIG | src_ips=
      [2025-11-02 10:02:04.930233][BSO] CONFIG | dst_ips=
      [2025-11-02 10:02:04.930234][BSO] CONFIG | src_ports=
      [2025-11-02 10:02:04.930235][BSO] CONFIG | dst_ports=
      [2025-11-02 10:02:04.930236][BSO] CONFIG | vlan_ids=
      [2025-11-02 10:02:04.930237][BSO] CONFIG | use_protocol_flows=0
      [2025-11-02 10:02:04.930238][BSO] CONFIG | packet_protocol=UDP
      [2025-11-02 10:02:04.930242][BSO] CONFIG | stream_mode=continuous
      [2025-11-02 10:02:04.930243][BSO] CONFIG | enable_segment_monitor=False
      [2025-11-02 10:02:04.930244][BSO] CONFIG | teaching_warmup_packet_type=generic
      [2025-11-02 10:02:04.930245][BSO] CONFIG | teaching_measurement_packet_type=
      [2025-11-02 10:02:04.930248][BSO] CONFIG | use_device_stats=True
      [2025-11-02 10:02:04.930249][BSO] CONFIG | send_teaching_warmup=True
      [2025-11-02 10:02:04.930250][BSO] CONFIG | send_teaching_measurement=True
      [2025-11-02 10:02:04.930252][BSO] CONFIG | trex_host=localhost
      [2025-11-02 10:02:04.930253][BSO] CONFIG | device_pairs=0:1
      [2025-11-02 10:02:04.930254][BSO] CONFIG | active_device_pairs=0:1
      [2025-11-02 10:02:04.930255][BSO] CONFIG | rate_unit=%
      [2025-11-02 10:02:04.930256][BSO] CONFIG | measure_latency=1
      [2025-11-02 10:02:04.930257][BSO] CONFIG | latency_rate=1000
      [2025-11-02 10:02:04.930258][BSO] CONFIG | enable_flow_cache=True
      [2025-11-02 10:02:04.930259][BSO] CONFIG | teaching_measurement_interval=10.0
      [2025-11-02 10:02:04.930260][BSO] CONFIG | teaching_warmup_packet_rate=10000
      [2025-11-02 10:02:04.930261][BSO] CONFIG | teaching_measurement_packet_rate=1000
      [2025-11-02 10:02:04.930262][BSO] CONFIG | no_promisc=False
      [2025-11-02 10:02:04.930263][BSO] CONFIG | latency_device_pair=--
      [2025-11-02 10:02:04.930372][BSO] querying TRex...
      [2025-11-02 10:02:04.930374][BSO] cmd: python3 -u /var/opt/trafficgen/./trex-query.py --trex-host=localhost --mirrored-log --device 0 --device 1
      [2025-11-02 10:02:05.136221][PQO] Namespace(trex_host='localhost', mirrored_log=True, device=[0, 1])
      [2025-11-02 10:02:05.136255][PQO] Establishing connection to TRex server...
      [2025-11-02 10:02:05.160393][PQO] Connection established
      [2025-11-02 10:02:05.186979][PQO] READABLE PORT INFO:
      [2025-11-02 10:02:05.187129][PQO] [
      [2025-11-02 10:02:05.187192][PQO]     {
      [2025-11-02 10:02:05.187212][PQO]         "cores": [
      [2025-11-02 10:02:05.187228][PQO]             0,
      [2025-11-02 10:02:05.187244][PQO]             1,
      [2025-11-02 10:02:05.187265][PQO]             2,
      [2025-11-02 10:02:05.187284][PQO]             3,
      [2025-11-02 10:02:05.187299][PQO]             4,
      [2025-11-02 10:02:05.187315][PQO]             5,
      [2025-11-02 10:02:05.187330][PQO]             6,
      [2025-11-02 10:02:05.187345][PQO]             7,
      [2025-11-02 10:02:05.187360][PQO]             8,
      [2025-11-02 10:02:05.187375][PQO]             9,
      [2025-11-02 10:02:05.187391][PQO]             10,
      [2025-11-02 10:02:05.187405][PQO]             11,
      [2025-11-02 10:02:05.187420][PQO]             12,
      [2025-11-02 10:02:05.187435][PQO]             13,
      [2025-11-02 10:02:05.187455][PQO]             14,
      [2025-11-02 10:02:05.187470][PQO]             15,
      [2025-11-02 10:02:05.187484][PQO]             16,
      [2025-11-02 10:02:05.187503][PQO]             17,
      [2025-11-02 10:02:05.187571][PQO]             18,
      [2025-11-02 10:02:05.187587][PQO]             19,
      [2025-11-02 10:02:05.187597][PQO]             20,
      [2025-11-02 10:02:05.187604][PQO]             21,
      [2025-11-02 10:02:05.187612][PQO]             22
      [2025-11-02 10:02:05.187619][PQO]         ],
      [2025-11-02 10:02:05.187633][PQO]         "description": "Ethernet Controller XXV710 for 25GbE SFP28",
      [2025-11-02 10:02:05.187650][PQO]         "driver": "net_i40e",
      [2025-11-02 10:02:05.187673][PQO]         "hw_mac": "3c:fd:fe:ad:bf:c4",
      [2025-11-02 10:02:05.187690][PQO]         "index": 0,
      [2025-11-02 10:02:05.187706][PQO]         "is_fc_supported": false,
      [2025-11-02 10:02:05.187721][PQO]         "is_led_supported": true,
      [2025-11-02 10:02:05.187737][PQO]         "is_link_supported": true,
      [2025-11-02 10:02:05.187752][PQO]         "is_prom_supported": true,
      [2025-11-02 10:02:05.187768][PQO]         "is_virtual": "no",
      [2025-11-02 10:02:05.187783][PQO]         "is_vxlan_supported": "yes",
      [2025-11-02 10:02:05.187804][PQO]         "numa": 0,
      [2025-11-02 10:02:05.187825][PQO]         "pci_addr": "0000:04:00.0",
      [2025-11-02 10:02:05.187843][PQO]         "rx": {
      [2025-11-02 10:02:05.187858][PQO]             "caps": [
      [2025-11-02 10:02:05.187877][PQO]                 "flow_stats",
      [2025-11-02 10:02:05.187892][PQO]                 "latency"
      [2025-11-02 10:02:05.187908][PQO]             ],
      [2025-11-02 10:02:05.187923][PQO]             "counters": 127
      [2025-11-02 10:02:05.187938][PQO]         },
      [2025-11-02 10:02:05.187953][PQO]         "stack": "legacy",
      [2025-11-02 10:02:05.187969][PQO]         "supp_speeds": [
      [2025-11-02 10:02:05.187991][PQO]             25000
      [2025-11-02 10:02:05.188006][PQO]         ],
      [2025-11-02 10:02:05.188020][PQO]         "status": "IDLE",
      [2025-11-02 10:02:05.188036][PQO]         "link": "UP",
      [2025-11-02 10:02:05.188050][PQO]         "fc": "none",
      [2025-11-02 10:02:05.188066][PQO]         "prom": "off",
      [2025-11-02 10:02:05.188081][PQO]         "mult": "off",
      [2025-11-02 10:02:05.188097][PQO]         "vxlan_fs": "-",
      [2025-11-02 10:02:05.188113][PQO]         "fc_supported": "no",
      [2025-11-02 10:02:05.188129][PQO]         "prom_supported": "yes",
      [2025-11-02 10:02:05.188140][PQO]         "led_change_supported": "yes",
      [2025-11-02 10:02:05.188148][PQO]         "link_change_supported": "yes",
      [2025-11-02 10:02:05.188157][PQO]         "speed": 25.0,
      [2025-11-02 10:02:05.188174][PQO]         "vlan": "-",
      [2025-11-02 10:02:05.188189][PQO]         "rx_filter_mode": "hardware match",
      [2025-11-02 10:02:05.188204][PQO]         "src_mac": "00:00:00:00:00:01",
      [2025-11-02 10:02:05.188226][PQO]         "layer_mode": "Ethernet",
      [2025-11-02 10:02:05.188242][PQO]         "src_ipv4": "-",
      [2025-11-02 10:02:05.188257][PQO]         "dest": "00:00:00:00:00:02",
      [2025-11-02 10:02:05.188274][PQO]         "arp": "-",
      [2025-11-02 10:02:05.188312][PQO]         "ipv6": "off",
      [2025-11-02 10:02:05.188328][PQO]         "rx_queue": "off",
      [2025-11-02 10:02:05.188344][PQO]         "grat_arp": "off"
      [2025-11-02 10:02:05.188365][PQO]     },
      [2025-11-02 10:02:05.188381][PQO]     {
      [2025-11-02 10:02:05.188397][PQO]         "cores": [
      [2025-11-02 10:02:05.188412][PQO]             0,
      [2025-11-02 10:02:05.188464][PQO]             1,
      [2025-11-02 10:02:05.188524][PQO]             2,
      [2025-11-02 10:02:05.188545][PQO]             3,
      [2025-11-02 10:02:05.188561][PQO]             4,
      [2025-11-02 10:02:05.188576][PQO]             5,
      [2025-11-02 10:02:05.188591][PQO]             6,
      [2025-11-02 10:02:05.188604][PQO]             7,
      [2025-11-02 10:02:05.188619][PQO]             8,
      [2025-11-02 10:02:05.188635][PQO]             9,
      [2025-11-02 10:02:05.188651][PQO]             10,
      [2025-11-02 10:02:05.188667][PQO]             11,
      [2025-11-02 10:02:05.188683][PQO]             12,
      [2025-11-02 10:02:05.188704][PQO]             13,
      [2025-11-02 10:02:05.188722][PQO]             14,
      [2025-11-02 10:02:05.188738][PQO]             15,
      [2025-11-02 10:02:05.188753][PQO]             16,
      [2025-11-02 10:02:05.188770][PQO]             17,
      [2025-11-02 10:02:05.188785][PQO]             18,
      [2025-11-02 10:02:05.188807][PQO]             19,
      [2025-11-02 10:02:05.188827][PQO]             20,
      [2025-11-02 10:02:05.188842][PQO]             21,
      [2025-11-02 10:02:05.188858][PQO]             22
      [2025-11-02 10:02:05.188874][PQO]         ],
      [2025-11-02 10:02:05.188890][PQO]         "description": "Ethernet Controller XXV710 for 25GbE SFP28",
      [2025-11-02 10:02:05.188906][PQO]         "driver": "net_i40e",
      [2025-11-02 10:02:05.188922][PQO]         "hw_mac": "3c:fd:fe:ad:bf:c5",
      [2025-11-02 10:02:05.188937][PQO]         "index": 1,
      [2025-11-02 10:02:05.188953][PQO]         "is_fc_supported": false,
      [2025-11-02 10:02:05.188969][PQO]         "is_led_supported": true,
      [2025-11-02 10:02:05.188985][PQO]         "is_link_supported": true,
      [2025-11-02 10:02:05.189002][PQO]         "is_prom_supported": true,
      [2025-11-02 10:02:05.189028][PQO]         "is_virtual": "no",
      [2025-11-02 10:02:05.189078][PQO]         "is_vxlan_supported": "yes",
      [2025-11-02 10:02:05.189094][PQO]         "numa": 0,
      [2025-11-02 10:02:05.189110][PQO]         "pci_addr": "0000:04:00.1",
      [2025-11-02 10:02:05.189131][PQO]         "rx": {
      [2025-11-02 10:02:05.189151][PQO]             "caps": [
      [2025-11-02 10:02:05.189169][PQO]                 "flow_stats",
      [2025-11-02 10:02:05.189187][PQO]                 "latency"
      [2025-11-02 10:02:05.189205][PQO]             ],
      [2025-11-02 10:02:05.189224][PQO]             "counters": 127
      [2025-11-02 10:02:05.189242][PQO]         },
      [2025-11-02 10:02:05.189268][PQO]         "stack": "legacy",
      [2025-11-02 10:02:05.189300][PQO]         "supp_speeds": [
      [2025-11-02 10:02:05.189318][PQO]             25000
      [2025-11-02 10:02:05.189337][PQO]         ],
      [2025-11-02 10:02:05.189358][PQO]         "status": "IDLE",
      [2025-11-02 10:02:05.189376][PQO]         "link": "UP",
      [2025-11-02 10:02:05.189394][PQO]         "fc": "none",
      [2025-11-02 10:02:05.189413][PQO]         "prom": "off",
      [2025-11-02 10:02:05.189432][PQO]         "mult": "off",
      [2025-11-02 10:02:05.189451][PQO]         "vxlan_fs": "-",
      [2025-11-02 10:02:05.189469][PQO]         "fc_supported": "no",
      [2025-11-02 10:02:05.189488][PQO]         "prom_supported": "yes",
      [2025-11-02 10:02:05.189507][PQO]         "led_change_supported": "yes",
      [2025-11-02 10:02:05.189527][PQO]         "link_change_supported": "yes",
      [2025-11-02 10:02:05.189545][PQO]         "speed": 25.0,
      [2025-11-02 10:02:05.189562][PQO]         "vlan": "-",
      [2025-11-02 10:02:05.189589][PQO]         "rx_filter_mode": "hardware match",
      [2025-11-02 10:02:05.189608][PQO]         "src_mac": "00:00:00:00:00:02",
      [2025-11-02 10:02:05.189627][PQO]         "layer_mode": "Ethernet",
      [2025-11-02 10:02:05.189644][PQO]         "src_ipv4": "-",
      [2025-11-02 10:02:05.189663][PQO]         "dest": "00:00:00:00:00:01",
      [2025-11-02 10:02:05.189686][PQO]         "arp": "-",
      [2025-11-02 10:02:05.189703][PQO]         "ipv6": "off",
      [2025-11-02 10:02:05.189721][PQO]         "rx_queue": "off",
      [2025-11-02 10:02:05.189738][PQO]         "grat_arp": "off"
      [2025-11-02 10:02:05.189759][PQO]     }
      [2025-11-02 10:02:05.189777][PQO] ]
      [2025-11-02 10:02:05.208156][PQO] Disconnecting from TRex server...
      [2025-11-02 10:02:05.208177][PQO] Connection severed
      [2025-11-02 10:02:05.208552][BSO] return code: 0
      [2025-11-02 10:02:05.208640][BSO] Starting binary-search
      [2025-11-02 10:02:05.208643][BSO] 
      [2025-11-02 10:02:05.208643][BSO] Trial Mode: Warmup
      [2025-11-02 10:02:05.208661][BSO] running trial 001, rate 25.0%
      [2025-11-02 10:02:05.208829][BSO] cmd: python3 -u /var/opt/trafficgen/./trex-txrx.py --trex-host=localhost --device-pairs=0:1 --active-device-pairs=0:1 --mirrored-log --measure-latency=1 --latency-rate=1000 --rate=25.0 --rate-unit=% --size=64 --runtime=10 --runtime-tolerance=10.0 --run-bidirec=1 --run-revunidirec=0 --num-flows=1024 --use-src-ip-flows=1 --use-dst-ip-flows=1 --use-src-mac-flows=1 --use-dst-mac-flows=0 --use-src-port-flows=0 --use-dst-port-flows=0 --use-protocol-flows=0 --packet-protocol=UDP --stream-mode=continuous --max-loss-pct=0.0 --skip-hw-flow-stats --send-teaching-warmup --send-teaching-measurement --teaching-measurement-interval=10.0 --teaching-warmup-packet-rate=10000 --teaching-measurement-packet-rate=1000 --teaching-warmup-packet-type=generic
      [2025-11-02 10:02:05.422198][001] Namespace(trex_host='localhost', debug=False, mirrored_log=True, frame_size='64', device_pairs='0:1', active_device_pairs='0:1', num_flows=1024, use_src_ip_flows=1, use_dst_ip_flows=1, use_src_mac_flows=1, use_dst_mac_flows=0, use_src_port_flows=0, use_dst_port_flows=0, use_protocol_flows=0, run_bidirec=1, run_revunidirec=0, runtime=10, runtime_tolerance=10.0, rate=25.0, rate_unit='%', packet_protocol='UDP', src_ports_list='', dst_ports_list='', dst_macs_list='', src_macs_list='', dst_ips_list='', src_ips_list='', vlan_ids_list='', measure_latency=1, latency_rate=1000, stream_mode='continuous', skip_hw_flow_stats=True, enable_segment_monitor=False, max_loss_pct=0.0, enable_flow_cache=True, send_teaching_warmup=True, send_teaching_measurement=True, teaching_measurement_interval=10.0, teaching_warmup_packet_rate=10000, teaching_measurement_packet_rate=1000, teaching_warmup_packet_type='generic', teaching_measurement_packet_type='garp', no_promisc=False, binary_search_synchronize=False, flow_mods={'mac': {'src': 1, 'dst': 0}, 'ip': {'src': 1, 'dst': 1}, 'port': {'src': 0, 'dst': 0}, 'protocol': 0})
      [2025-11-02 10:02:05.422242][001] Configuring device pair: 0<->1
      [2025-11-02 10:02:05.423764][001] Active TX Ports: 2
      [2025-11-02 10:02:05.544242][001] Establishing connection to TRex server...
      [2025-11-02 10:02:05.544270][001] Connection established
      [2025-11-02 10:02:05.581294][001] Creating stream for device pair '0:1' direction '->' with packet_type=[default], protocol=[UDP], pps=[9300595.238095], pg_id=[1], name=[default_stream_0->1_UDP], frame_size=[64], next_stream_name=[None], self_start=[True], stream_mode=[burst], runtime=[10.000000], and traffic_share=[1.000000].
      [2025-11-02 10:02:05.582071][001] Creating stream for device pair '0:1' direction '->' with packet_type=[latency], protocol=[UDP], pps=[1000.000000], pg_id=[64], name=[latency_stream_0->1_UDP], frame_size=[64], next_stream_name=[None], self_start=[True], stream_mode=[continuous], runtime=[-1.000000], and traffic_share=[1.000000].
      [2025-11-02 10:02:05.585155][001] Creating teaching warmup streams for device pair '0:1' direction '->' with TYPE=generic and MAC=00:00:00:00:00:02 and IP=2.2.2.2
      [2025-11-02 10:02:05.586325][001] Creating teaching measurement streams for device pair '0:1' direction '->' with TYPE=garp and MAC=00:00:00:00:00:02 and IP=2.2.2.2
      [2025-11-02 10:02:05.588230][001] Creating stream for device pair '0:1' direction '<-' with packet_type=[default], protocol=[UDP], pps=[9300595.238095], pg_id=[128], name=[default_stream_1->0_UDP], frame_size=[64], next_stream_name=[None], self_start=[True], stream_mode=[burst], runtime=[10.000000], and traffic_share=[1.000000].
      [2025-11-02 10:02:05.589976][001] Creating stream for device pair '0:1' direction '<-' with packet_type=[latency], protocol=[UDP], pps=[1000.000000], pg_id=[191], name=[latency_stream_1->0_UDP], frame_size=[64], next_stream_name=[None], self_start=[True], stream_mode=[continuous], runtime=[-1.000000], and traffic_share=[1.000000].
      [2025-11-02 10:02:05.592320][001] Creating teaching warmup streams for device pair '0:1' direction '<-' with TYPE=generic and MAC=00:00:00:00:00:01 and IP=1.1.1.1
      [2025-11-02 10:02:05.594198][001] Creating teaching measurement streams for device pair '0:1' direction '<-' with TYPE=garp and MAC=00:00:00:00:00:01 and IP=1.1.1.1
      [2025-11-02 10:02:05.595889][001] Adding teaching warmup stream(s) for device pair '0:1' to port 0
      [2025-11-02 10:02:05.597238][001] Adding teaching warmup stream(s) for device pair '0:1' to port 1
      [2025-11-02 10:02:05.598491][001] Transmitting teaching warmup packets...
      [2025-11-02 10:02:05.713303][001] ...teaching warmup transmission complete -- 0 total second(s) elapsed
      [2025-11-02 10:02:05.740002][001] Adding stream(s) for device pair '0:1' to port 0
      [2025-11-02 10:02:05.740027][001] Adding teaching stream(s) for device pair '0:1' to port 0
      [2025-11-02 10:02:05.745339][001] Adding stream(s) for device pair '0:1' to port 1
      [2025-11-02 10:02:05.752558][001] Adding teaching stream(s) for device pair '0:1' to port 1
      [2025-11-02 10:02:05.752583][001] Starting test at 10:02:05 on 2025-11-02
      [2025-11-02 10:02:05.752598][001] The test should end at 10:02:15 on 2025-11-02
      [2025-11-02 10:02:05.752613][001] The test will timeout with an error at 10:02:16 on 2025-11-02
      [2025-11-02 10:02:05.757555][001] Transmitting at 25.0%% from port 0 to port 1 for 10 seconds...
      [2025-11-02 10:02:05.757574][001] Transmitting at 25.0%% from port 1 to port 0 for 10 seconds...
      [2025-11-02 10:02:05.757593][001] Waiting...
      [2025-11-02 10:02:15.779801][001] Finished test at 10:02:15 on 2025-11-02
      [2025-11-02 10:02:15.779824][001] Test ran for 10 seconds (0:00:10.027245)
      [2025-11-02 10:02:15.782748][001] TRex Events:
      [2025-11-02 10:02:15.782771][001]     EVENT: 2025-11-02 10:02:05 - [server][info]     - Profile 0._ has started
      [2025-11-02 10:02:15.782785][001]     EVENT: 2025-11-02 10:02:05 - [server][info]     - Port 0 has started
      [2025-11-02 10:02:15.782800][001]     EVENT: 2025-11-02 10:02:05 - [server][info]     - Profile 1._ has started
      [2025-11-02 10:02:15.782812][001]     EVENT: 2025-11-02 10:02:05 - [server][info]     - Port 1 has started
      [2025-11-02 10:02:15.782824][001]     EVENT: 2025-11-02 10:02:15 - [server][info]     - Profile 0._ job done
      [2025-11-02 10:02:15.782839][001]     EVENT: 2025-11-02 10:02:15 - [server][info]     - Port 0 job done
      [2025-11-02 10:02:15.782851][001]     EVENT: 2025-11-02 10:02:15 - [server][info]     - Profile 1._ job done
      [2025-11-02 10:02:15.782862][001]     EVENT: 2025-11-02 10:02:15 - [server][info]     - Port 1 job done
      [2025-11-02 10:02:15.782877][001] TX Utilization: 5.697158%
      [2025-11-02 10:02:15.783787][001] RX Utilization: 0.000000%
      [2025-11-02 10:02:15.783802][001] TX Queue Full:  0
      [2025-11-02 10:02:15.783819][001] Disconnecting from TRex server...
      [2025-11-02 10:02:15.787818][BSO] Device Pair: 0->1 |   All TX   | packets=93,018,000 rate=9,276,526.10462794 l1_bps=6,233,825,542.309976 l2_bps=4,452,732,530.221412
      [2025-11-02 10:02:15.787828][BSO] Device Pair: 0->1 |   All RX   | packets=0 rate=0.0 l1_bps=0.0 l2_bps=0.0
      [2025-11-02 10:02:15.787902][BSO] Device Pair: 0->1 | Latency TX | packets=10,000 rate=997.282902731508 l1_bps=670,174.1106355734 l2_bps=478,695.7933111238
      [2025-11-02 10:02:15.787915][BSO] Device Pair: 0->1 | Latency RX | packets=0 rate=0.0 l1_bps=0.0 l2_bps=0.0 average=-1.0 minimum=0.0 maximum=0.0
      [2025-11-02 10:02:15.787953][BSO] Device Pair: 1->0 |   All TX   | packets=93,018,000 rate=9,276,526.10462794 l1_bps=6,233,825,542.309976 l2_bps=4,452,732,530.221412
      [2025-11-02 10:02:15.787961][BSO] Device Pair: 1->0 |   All RX   | packets=0 rate=0.0 l1_bps=0.0 l2_bps=0.0
      [2025-11-02 10:02:15.788017][BSO] Device Pair: 1->0 | Latency TX | packets=10,000 rate=997.282902731508 l1_bps=670,174.1106355734 l2_bps=478,695.7933111238
      [2025-11-02 10:02:15.788029][BSO] Device Pair: 1->0 | Latency RX | packets=0 rate=0.0 l1_bps=0.0 l2_bps=0.0 average=-1.0 minimum=0.0 maximum=0.0
      [2025-11-02 10:02:15.788085][001] Connection severed
      [2025-11-02 10:02:15.807072][BSO] tg return code: 0
      [2025-11-02 10:02:15.807076][BSO] Traffic generator set trial start time
      [2025-11-02 10:02:15.807077][BSO] Traffic generator set trial stop time
      [2025-11-02 10:02:15.807138][BSO] (Evaluating trial)
      [2025-11-02 10:02:15.807144][BSO] 	(critical requirement failure, no packets were received between device pair: 0 -> 1, trial result: abort)
      [2025-11-02 10:02:15.807149][BSO] 	(critical requirement failure, individual stream 100% RX packet loss , device pair: 0 -> 1, pg_ids: [64], trial result: abort)
      [2025-11-02 10:02:15.807152][BSO] 	(critical requirement failure, no packets were received between device pair: 1 -> 0, trial result: abort)
      [2025-11-02 10:02:15.807154][BSO] 	(critical requirement failure, individual stream 100% RX packet loss , device pair: 1 -> 0, pg_ids: [191], trial result: abort)
      [2025-11-02 10:02:15.807158][BSO] 	(since this was a warmup trial the actual results do not matter and are ignored, trial result status: modified, trial result: pass)
      [2025-11-02 10:02:15.807176][BSO] 	(trial passed requirement, TX rate tolerance, device pair: 0 -> 1, unit: mpps, tolerance: 8.370535714285715 - 10.230654761904765, achieved: 9.27652610462794, trial result status: unmodified, trial result: pass)
      [2025-11-02 10:02:15.807183][BSO] 	(trial passed requirement, TX rate tolerance, device pair: 1 -> 0, unit: mpps, tolerance: 8.370535714285715 - 10.230654761904765, achieved: 9.27652610462794, trial result status: unmodified, trial result: pass)
      [2025-11-02 10:02:15.807195][BSO] 	(trial failed requirement, percent loss, direction: ->, requested: 0.0%, achieved: 100.0%, lost packets: 93,018,000, trial result status: modified, trial result: fail)
      [2025-11-02 10:02:15.807199][BSO] 	(trial failed requirement, percent loss, direction: <-, requested: 0.0%, achieved: 100.0%, lost packets: 93,018,000, trial result status: modified, trial result: fail)
      [2025-11-02 10:02:15.807205][BSO] 	(since this was a warmup trial the actual results do not matter and are ignored, trial result status: modified, trial result: pass)
      [2025-11-02 10:02:15.807696][BSO] (trial results are ignored since this is a warmup run)
      [2025-11-02 10:02:15.807701][BSO] 
      [2025-11-02 10:02:15.807701][BSO] Trial Mode: Search
      [2025-11-02 10:02:15.807711][BSO] running trial 002, rate 25.0%
      [2025-11-02 10:02:15.807827][BSO] cmd: python3 -u /var/opt/trafficgen/./trex-txrx.py --trex-host=localhost --device-pairs=0:1 --active-device-pairs=0:1 --mirrored-log --measure-latency=1 --latency-rate=1000 --rate=25.0 --rate-unit=% --size=64 --runtime=10 --runtime-tolerance=10.0 --run-bidirec=1 --run-revunidirec=0 --num-flows=1024 --use-src-ip-flows=1 --use-dst-ip-flows=1 --use-src-mac-flows=1 --use-dst-mac-flows=0 --use-src-port-flows=0 --use-dst-port-flows=0 --use-protocol-flows=0 --packet-protocol=UDP --stream-mode=continuous --max-loss-pct=0.0 --skip-hw-flow-stats --send-teaching-warmup --send-teaching-measurement --teaching-measurement-interval=10.0 --teaching-warmup-packet-rate=10000 --teaching-measurement-packet-rate=1000 --teaching-warmup-packet-type=generic
      [2025-11-02 10:02:16.023186][002] Namespace(trex_host='localhost', debug=False, mirrored_log=True, frame_size='64', device_pairs='0:1', active_device_pairs='0:1', num_flows=1024, use_src_ip_flows=1, use_dst_ip_flows=1, use_src_mac_flows=1, use_dst_mac_flows=0, use_src_port_flows=0, use_dst_port_flows=0, use_protocol_flows=0, run_bidirec=1, run_revunidirec=0, runtime=10, runtime_tolerance=10.0, rate=25.0, rate_unit='%', packet_protocol='UDP', src_ports_list='', dst_ports_list='', dst_macs_list='', src_macs_list='', dst_ips_list='', src_ips_list='', vlan_ids_list='', measure_latency=1, latency_rate=1000, stream_mode='continuous', skip_hw_flow_stats=True, enable_segment_monitor=False, max_loss_pct=0.0, enable_flow_cache=True, send_teaching_warmup=True, send_teaching_measurement=True, teaching_measurement_interval=10.0, teaching_warmup_packet_rate=10000, teaching_measurement_packet_rate=1000, teaching_warmup_packet_type='generic', teaching_measurement_packet_type='garp', no_promisc=False, binary_search_synchronize=False, flow_mods={'mac': {'src': 1, 'dst': 0}, 'ip': {'src': 1, 'dst': 1}, 'port': {'src': 0, 'dst': 0}, 'protocol': 0})
      [2025-11-02 10:02:16.023226][002] Configuring device pair: 0<->1
      [2025-11-02 10:02:16.023448][002] Active TX Ports: 2
      [2025-11-02 10:02:16.023471][002] Establishing connection to TRex server...
      [2025-11-02 10:02:16.147394][002] Connection established
      [2025-11-02 10:02:16.195196][002] Creating stream for device pair '0:1' direction '->' with packet_type=[default], protocol=[UDP], pps=[9300595.238095], pg_id=[1], name=[default_stream_0->1_UDP], frame_size=[64], next_stream_name=[None], self_start=[True], stream_mode=[burst], runtime=[10.000000], and traffic_share=[1.000000].
      [2025-11-02 10:02:16.195848][002] Creating stream for device pair '0:1' direction '->' with packet_type=[latency], protocol=[UDP], pps=[1000.000000], pg_id=[64], name=[latency_stream_0->1_UDP], frame_size=[64], next_stream_name=[None], self_start=[True], stream_mode=[continuous], runtime=[-1.000000], and traffic_share=[1.000000].
      [2025-11-02 10:02:16.198980][002] Creating teaching warmup streams for device pair '0:1' direction '->' with TYPE=generic and MAC=00:00:00:00:00:02 and IP=2.2.2.2
      [2025-11-02 10:02:16.200100][002] Creating teaching measurement streams for device pair '0:1' direction '->' with TYPE=garp and MAC=00:00:00:00:00:02 and IP=2.2.2.2
      [2025-11-02 10:02:16.201896][002] Creating stream for device pair '0:1' direction '<-' with packet_type=[default], protocol=[UDP], pps=[9300595.238095], pg_id=[128], name=[default_stream_1->0_UDP], frame_size=[64], next_stream_name=[None], self_start=[True], stream_mode=[burst], runtime=[10.000000], and traffic_share=[1.000000].
      [2025-11-02 10:02:16.203725][002] Creating stream for device pair '0:1' direction '<-' with packet_type=[latency], protocol=[UDP], pps=[1000.000000], pg_id=[191], name=[latency_stream_1->0_UDP], frame_size=[64], next_stream_name=[None], self_start=[True], stream_mode=[continuous], runtime=[-1.000000], and traffic_share=[1.000000].
      [2025-11-02 10:02:16.205650][002] Creating teaching warmup streams for device pair '0:1' direction '<-' with TYPE=generic and MAC=00:00:00:00:00:01 and IP=1.1.1.1
      [2025-11-02 10:02:16.207875][002] Creating teaching measurement streams for device pair '0:1' direction '<-' with TYPE=garp and MAC=00:00:00:00:00:01 and IP=1.1.1.1
      [2025-11-02 10:02:16.209689][002] Adding teaching warmup stream(s) for device pair '0:1' to port 0
      [2025-11-02 10:02:16.211189][002] Adding teaching warmup stream(s) for device pair '0:1' to port 1
      [2025-11-02 10:02:16.212682][002] Transmitting teaching warmup packets...
      [2025-11-02 10:02:16.327614][002] ...teaching warmup transmission complete -- 0 total second(s) elapsed
      [2025-11-02 10:02:16.356384][002] Adding stream(s) for device pair '0:1' to port 0
      [2025-11-02 10:02:16.356408][002] Adding teaching stream(s) for device pair '0:1' to port 0
      [2025-11-02 10:02:16.362250][002] Adding stream(s) for device pair '0:1' to port 1
      [2025-11-02 10:02:16.370914][002] Adding teaching stream(s) for device pair '0:1' to port 1
      [2025-11-02 10:02:16.370941][002] Starting test at 10:02:16 on 2025-11-02
      [2025-11-02 10:02:16.370955][002] The test should end at 10:02:26 on 2025-11-02
      [2025-11-02 10:02:16.370968][002] The test will timeout with an error at 10:02:27 on 2025-11-02
      [2025-11-02 10:02:16.370984][002] Transmitting at 25.0%% from port 0 to port 1 for 10 seconds...
      [2025-11-02 10:02:16.376303][002] Transmitting at 25.0%% from port 1 to port 0 for 10 seconds...
      [2025-11-02 10:02:16.376329][002] Waiting...
      [2025-11-02 10:02:26.398345][002] Finished test at 10:02:26 on 2025-11-02
      [2025-11-02 10:02:26.398369][002] Test ran for 10 seconds (0:00:10.027433)
      [2025-11-02 10:02:26.401486][002] TRex Events:
      [2025-11-02 10:02:26.401510][002]     EVENT: 2025-11-02 10:02:16 - [server][info]     - Profile 0._ has started
      [2025-11-02 10:02:26.401524][002]     EVENT: 2025-11-02 10:02:16 - [server][info]     - Port 0 has started
      [2025-11-02 10:02:26.401540][002]     EVENT: 2025-11-02 10:02:16 - [server][info]     - Profile 1._ has started
      [2025-11-02 10:02:26.401553][002]     EVENT: 2025-11-02 10:02:16 - [server][info]     - Port 1 has started
      [2025-11-02 10:02:26.401569][002]     EVENT: 2025-11-02 10:02:26 - [server][info]     - Profile 0._ job done
      [2025-11-02 10:02:26.401582][002]     EVENT: 2025-11-02 10:02:26 - [server][info]     - Port 0 job done
      [2025-11-02 10:02:26.401596][002]     EVENT: 2025-11-02 10:02:26 - [server][info]     - Profile 1._ job done
      [2025-11-02 10:02:26.401609][002]     EVENT: 2025-11-02 10:02:26 - [server][info]     - Port 1 job done
      [2025-11-02 10:02:26.401622][002] TX Utilization: 5.741618%
      [2025-11-02 10:02:26.401637][002] RX Utilization: 0.000000%
      [2025-11-02 10:02:26.402518][002] TX Queue Full:  0
      [2025-11-02 10:02:26.402550][002] Disconnecting from TRex server...
      [2025-11-02 10:02:26.404150][002] Connection severed
      [2025-11-02 10:02:26.407052][BSO] Device Pair: 0->1 |   All TX   | packets=93,018,001 rate=9,276,352.282782642 l1_bps=6,233,708,734.029936 l2_bps=4,452,649,095.735668
      [2025-11-02 10:02:26.407061][BSO] Device Pair: 0->1 |   All RX   | packets=0 rate=0.0 l1_bps=0.0 l2_bps=0.0
      [2025-11-02 10:02:26.407137][BSO] Device Pair: 0->1 | Latency TX | packets=10,001 rate=997.3639315266429 l1_bps=670,228.561985904 l2_bps=478,734.6871327886
      [2025-11-02 10:02:26.407150][BSO] Device Pair: 0->1 | Latency RX | packets=0 rate=0.0 l1_bps=0.0 l2_bps=0.0 average=-1.0 minimum=0.0 maximum=0.0
      [2025-11-02 10:02:26.407186][BSO] Device Pair: 1->0 |   All TX   | packets=93,018,001 rate=9,276,352.282782642 l1_bps=6,233,708,734.029936 l2_bps=4,452,649,095.735668
      [2025-11-02 10:02:26.407194][BSO] Device Pair: 1->0 |   All RX   | packets=0 rate=0.0 l1_bps=0.0 l2_bps=0.0
      [2025-11-02 10:02:26.407250][BSO] Device Pair: 1->0 | Latency TX | packets=10,001 rate=997.3639315266429 l1_bps=670,228.561985904 l2_bps=478,734.6871327886
      [2025-11-02 10:02:26.407263][BSO] Device Pair: 1->0 | Latency RX | packets=0 rate=0.0 l1_bps=0.0 l2_bps=0.0 average=-1.0 minimum=0.0 maximum=0.0
      [2025-11-02 10:02:26.426266][BSO] tg return code: 0
      [2025-11-02 10:02:26.426269][BSO] Traffic generator set trial start time
      [2025-11-02 10:02:26.426270][BSO] Traffic generator set trial stop time
      [2025-11-02 10:02:26.426340][BSO] (Evaluating trial)
      [2025-11-02 10:02:26.426346][BSO] 	(critical requirement failure, no packets were received between device pair: 0 -> 1, trial result: abort)
      [2025-11-02 10:02:26.426352][BSO] 	(critical requirement failure, individual stream 100% RX packet loss , device pair: 0 -> 1, pg_ids: [64], trial result: abort)
      [2025-11-02 10:02:26.426355][BSO] 	(critical requirement failure, no packets were received between device pair: 1 -> 0, trial result: abort)
      [2025-11-02 10:02:26.426357][BSO] 	(critical requirement failure, individual stream 100% RX packet loss , device pair: 1 -> 0, pg_ids: [191], trial result: abort)
      [2025-11-02 10:02:26.426365][BSO] ERROR: (binary search aborting due to critical error, trial result: quit)
      FAIL: binary-search.py
      State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
      LISTEN 0      100          0.0.0.0:4500      0.0.0.0:*          
      LISTEN 0      100          0.0.0.0:4501      0.0.0.0:*          
      LISTEN 0      100          0.0.0.0:4507      0.0.0.0:*          
      root         356  0.0  0.0   4320  2804 ?        Ss   09:57   0:00 /bin/bash ./t-rex-64 --no-ofed-check -c 23 -i --iom 0
      root         437  194  0.0 69350428 34404 ?      SLl  09:57   8:52 ./_t-rex-64 --no-ofed-check -c 23 -i --iom 0
      root         505  0.0  0.0   4188  2768 ?        Ss   10:02   0:00 bash -c ps -aux | grep t-rex
      root         507  0.0  0.0   3332  1384 ?        S    10:02   0:00 grep t-rex
      

       Logs: If you collected logs please provide them (e.g. sos report, /var/log/openvswitch/* , testpmd console)

      demsg log on DUT: https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/11/118699/11869968/19906719/205115600/dmesg.log
      OVS log:
      https://beaker.engineering.redhat.com/recipes/19906719/tasks/205115600/logs/ovs-vswitchd.log
      dmesg log on Trex side:
      https://beaker-archive.prod.engineering.redhat.com/beaker-logs/2025/11/118699/11869968/19906720/205115603/dmesg.log
      DUT sosreport:
      http://netqe-infra01.knqe.eng.rdu2.dc.redhat.com/sosreports/sosreport-dell-per740-03-2025-11-02-qqhueoo.tar.xz
      Trex sosreport:
      http://netqe-infra01.knqe.eng.rdu2.dc.redhat.com/sosreports/sosreport-dell-per730-53-2025-11-02-cqevhwa.tar.xz

              ovsdpdk-triage ovsdpdk triage
              nstbot NST Bot
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: