Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-10600

[OCP 4.13] Telco DPDK performance degradation

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • 4.13
    • Node Tuning Operator
    • None

      Description of problem:

      Our telco-tuned dut pods are not starting in 4.13 with these definitions: 
      irq-load-balancing.crio.io: disable 
      cpu-load-balancing.crio.io: disable
      
      31s         Normal    Scheduled        pod/dpdk-testpmd   Successfully assigned default/dpdk-testpmd to ostest-zh6lq-worker-0
      31s         Normal    AddedInterface   pod/dpdk-testpmd   Add eth0 [10.131.0.44/23] from ovn-kubernetes
      21s         Normal    Pulled           pod/dpdk-testpmd   Container image "registry.redhat.io/openshift4/dpdk-base-rhel8:v4.10.0-5" already present on machine
      21s         Normal    Created          pod/dpdk-testpmd   Created container dpdk-testpmd
      11s         Warning   Failed           pod/dpdk-testpmd   Error: failed to run pre-start hook for container "dpdk-testpmd": set CPU load balancing: timed out waiting for the condition
      
      Removal of these components will result in a pod starting, but the performance will be significantly degraded.

      Actual results:

      Actual results:
      Per-port with packet loss - 0.0019
      Total Mpps - 0.0038

      Expected results:

      Expected results:
      Per-port with 0 packet loss - 1.87
      Total Mpps - 3.74

      Additional info:

      (shiftstack) [cloud-user@installer-host ~]$ oc get clusterversion
      NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.13.0-0.nightly-2023-03-14-053612   True        False         5h19m   Cluster version is 4.13.0-0.nightly-2023-03-14-053612
      
      core@ostest-zh6lq-worker-1 ~]$ cat /etc/os-release
      NAME="CentOS Stream CoreOS"
      ID="rhcos"
      ID_LIKE="rhel fedora"
      VERSION="413.92.202303061740-0"
      VERSION_ID="4.13"
      VARIANT="CoreOS"
      VARIANT_ID=coreos
      PLATFORM_ID="platform:el9"
      PRETTY_NAME="CentOS Stream CoreOS 413.92.202303061740-0 (Plow)"
      ANSI_COLOR="0;31"
      CPE_NAME="cpe:/o:centos:centos:9coreos"
      HOME_URL="https://centos.org/"
      DOCUMENTATION_URL="https://docs.openshift.com/container-platform/4.13/"
      BUG_REPORT_URL="https://bugzilla.redhat.com/"
      REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
      REDHAT_BUGZILLA_PRODUCT_VERSION="4.13"
      REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
      REDHAT_SUPPORT_PRODUCT_VERSION="4.13"
      OPENSHIFT_VERSION="4.13"
      RHEL_VERSION="9"
      OSTREE_VERSION="413.92.202303061740-0"

              yquinn@redhat.com Yanir Quinn
              zgreenbe@redhat.com Ziv Greenberg
              Mallapadi Niranjan Mallapadi Niranjan
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: