Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-31095

TuneD stderr: ERROR tuned.plugins.plugin_scheduler: Failed to set affinity of PID 1110 to ''[29]'': [Errno 22] Invalid argument'

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.14.z
    • Node Tuning Operator
    • None
    • Important
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

          Performing the hardware reboot on each MNO nodes the tuned enters the following state:
      NAMESPACE                                NAME                                              TUNED                APPLIED   DEGRADED   AGE
      openshift-cluster-node-tuning-operator   master0.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-master   True      False      10h
      openshift-cluster-node-tuning-operator   master1.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-master   True      False      12h
      openshift-cluster-node-tuning-operator   master2.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-master   True      False      11h
      openshift-cluster-node-tuning-operator   worker0.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-worker   True      True       11h
      openshift-cluster-node-tuning-operator   worker1.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-worker   True      False      11h
      

      Version-Release number of selected component (if applicable):

       NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.14.14   True        False         11h     Cluster version is 4.14.14
      

      How reproducible:

          80%

      Steps to Reproduce:

          1.Apply the PAO and Tuned patch to the cluster highlited in the following support case: https://access.redhat.com/support/cases/#/case/03708995/discussion?commentId=a0a6R00000WCpsBQAT
          2. Perform reboot of each individual node in the cluster for 24h
          3.
          

      Actual results:

      NAMESPACE                                NAME                                              TUNED                APPLIED   DEGRADED   AGE
      openshift-cluster-node-tuning-operator   master0.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-master   True      False      10h
      openshift-cluster-node-tuning-operator   master1.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-master   True      False      12h
      openshift-cluster-node-tuning-operator   master2.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-master   True      False      11h
      openshift-cluster-node-tuning-operator   worker0.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-worker   True      True       11h
      openshift-cluster-node-tuning-operator   worker1.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-worker   True      False      11h
      
      apiVersion: tuned.openshift.io/v1
      kind: Profile
      metadata:
        creationTimestamp: "2024-03-19T19:20:50Z"
        generation: 4
        name: worker0.winterfell-mno-2.lab.neat.nsn-rdnet.net
        namespace: openshift-cluster-node-tuning-operator
        ownerReferences:
        - apiVersion: tuned.openshift.io/v1
          blockOwnerDeletion: true
          controller: true
          kind: Tuned
          name: default
          uid: 5ab6d677-ef3e-4eaf-98c5-8aa3038c9f5f
        resourceVersion: "138405"
        uid: dedf4e04-a88c-4961-98a6-0b971653f5f2
      spec:
        config:
          debug: false
          providerName: baremetalhost
          tunedConfig: {}
          tunedProfile: rdpmc-patch-worker
      status:
        conditions:
        - lastTransitionTime: "2024-03-19T19:58:08Z"
          message: TuneD profile applied.
          reason: AsExpected
          status: "True"
          type: Applied
        - lastTransitionTime: "2024-03-19T21:34:23Z"
          message: 'TuneD daemon issued one or more error message(s) during profile application.
            TuneD stderr:  ERROR    tuned.plugins.plugin_scheduler: Failed to set affinity
            of PID 1110 to ''[29]'': [Errno 22] Invalid argument'
          reason: TunedError
          status: "True"
          type: Degraded
        tunedProfile: rdpmc-patch-worker $ omc logs -n openshift-cluster-node-tuning-operator tuned-4p4qm | grep ERROR
      2024-03-19T21:34:20.242693269Z 2024-03-19 21:34:20,242 ERROR    tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 476 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/476/smp_affinity'
      2024-03-19T21:34:20.242855820Z 2024-03-19 21:34:20,242 ERROR    tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 493 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/493/smp_affinity'
      2024-03-19T21:34:20.243051936Z 2024-03-19 21:34:20,243 ERROR    tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 510 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/510/smp_affinity'
      2024-03-19T21:34:20.243449807Z 2024-03-19 21:34:20,243 ERROR    tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 517 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/517/smp_affinity'
      2024-03-19T21:34:20.243654335Z 2024-03-19 21:34:20,243 ERROR    tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 519 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/519/smp_affinity'
      2024-03-19T21:34:20.244274428Z 2024-03-19 21:34:20,243 ERROR    tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 595 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/595/smp_affinity'
      2024-03-19T21:34:20.403920714Z 2024-03-19 21:34:20,403 ERROR    tuned.plugins.plugin_scheduler: Failed to set affinity of PID 1110 to '[29]': [Errno 22] Invalid argument
      2024-03-19T21:34:25.816277984Z 2024-03-19 21:34:25,816 ERROR    tuned.plugins.plugin_scheduler: Failed to set affinity of PID 1110 to '[29]': [Errno 22] Invalid argument
      

      Expected results:

      NAMESPACE                                NAME                                              TUNED                APPLIED   DEGRADED   AGE
      openshift-cluster-node-tuning-operator   master0.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-master   True      False      11h
      openshift-cluster-node-tuning-operator   master1.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-master   True      False      13h
      openshift-cluster-node-tuning-operator   master2.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-master   True      False      13h
      openshift-cluster-node-tuning-operator   worker0.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-worker   True      False      13h
      openshift-cluster-node-tuning-operator   worker1.winterfell-mno-2.lab.neat.nsn-rdnet.net   rdpmc-patch-worker   True      False      13h
      

      Additional info:

          

            team-nto Team NTO
            midu@redhat.com Mihai IDU
            Liquan Cui Liquan Cui
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: