Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-24638

Tuned Profiles going degraded due to the extra net.core.rps_default_mask configuration in openshift-node-performance-xxx-profile

    XMLWordPrintable

Details

    • No
    • CNF Compute Sprint 246
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      * The platform tries to configure the same `sysctl` value early at boot and later when the node tuning is applied by {product-title}. As a result of a tuned issue this confuses the error reporting logic and results in a false negative report. There is no bug on the node itself, just a wrong degradation report. With this release, the configuration was updated to prevent this double configuration. (link:https://issues.redhat.com/browse/OCPBUGS-24638[*OCPBUGS-24638*])
      Show
      * The platform tries to configure the same `sysctl` value early at boot and later when the node tuning is applied by {product-title}. As a result of a tuned issue this confuses the error reporting logic and results in a false negative report. There is no bug on the node itself, just a wrong degradation report. With this release, the configuration was updated to prevent this double configuration. (link: https://issues.redhat.com/browse/OCPBUGS-24638 [* OCPBUGS-24638 *])
    • Bug Fix
    • In Progress

    Description

      Description of problem:
      Issue - Profiles are degraded [1]even after applied due to below [2]error:

      [1]

      $oc get profile -A
NAMESPACE                                NAME                                          TUNED                APPLIED   DEGRADED   AGE
openshift-cluster-node-tuning-operator   master0    rdpmc-patch-master   True      True       5d
openshift-cluster-node-tuning-operator   master1    rdpmc-patch-master   True      True       5d
openshift-cluster-node-tuning-operator   master2    rdpmc-patch-master   True      True       5d
openshift-cluster-node-tuning-operator   worker0    rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker1    rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker10   rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker11   rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker12   rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker13   rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker14   rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker15   rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker2    rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker3    rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker4  rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker5    rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker6    rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker7    rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker8   rdpmc-patch-worker   True      True       5d
openshift-cluster-node-tuning-operator   worker9   rdpmc-patch-worker   True      True       5d

      [2]

        lastTransitionTime: "2023-12-05T22:43:12Z"
    message: TuneD daemon issued one or more sysctl override message(s) during profile
      application. Use reapply_sysctl=true or remove conflicting sysctl net.core.rps_default_mask
    reason: TunedSysctlOverride
    status: "True"

      If we see in rdpmc-patch-master tuned:

      NAMESPACE                                NAME                                          TUNED                APPLIED   DEGRADED   AGE
openshift-cluster-node-tuning-operator   master0    rdpmc-patch-master   True      True       5d
openshift-cluster-node-tuning-operator   master1    rdpmc-patch-master   True      True       5d
openshift-cluster-node-tuning-operator   master2    rdpmc-patch-master   True      True       5d

      We are configuring below in rdpmc-patch-master tuned:

      $ oc get tuned rdpmc-patch-master -n openshift-cluster-node-tuning-operator -oyaml |less
spec:
  profile:
  - data: |
      [main]
      include=performance-patch-master
      [sysfs]
      /sys/devices/cpu/rdpmc = 2
    name: rdpmc-patch-master
  recommend:

      Below in Performance-patch-master which is included in above tuned:

      spec:
  profile:
  - data: |
      [main]
      summary=Custom tuned profile to adjust performance
      include=openshift-node-performance-master-profile
      [bootloader]
      cmdline_removeKernelArgs=-nohz_full=${isolated_cores}

      Below(which is coming in error) is in openshift-node-performance-master-profile included in above tuned:

      net.core.rps_default_mask=${not_isolated_cpumask}

      RHEL BUg has been raised for the same https://issues.redhat.com/browse/RHEL-18972

          Version-Release number of selected component (if applicable):{code:none}
4.14

      Attachments

        Issue Links

          blocks

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-25092 Tuned Profiles going degraded due to the extra net.core.rps_default_mask configuration in openshift-node-performance-xxx-profile

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-25092 Tuned Profiles going degraded due to the extra net.core.rps_default_mask configuration in openshift-node-performance-xxx-profile

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-23167 On SNO with DU profile(RT kernel) tuned profile is always degraded due to net.core.busy_read, net.core.busy_poll and kernel.numa_balancing sysctl not existing in RT kernel

          • Undefined - Priority not specified.
          • Verified
          links to

          GitHub openshift/cluster-node-tuning-operator#868: OCPBUGS-24638: Do not set default RPS sysctl twice

          Web Link RHEA-2024:0041 OpenShift Container Platform 4.16.z bug fix update

          Activity

            People

              msivak@redhat.com Martin Sivak
              rhn-support-psingour Poornima Singour
              Mallapadi Niranjan Mallapadi Niranjan
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: