Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-36597

Error removing kernel args with tuned profile

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • 4.14
    • Node Tuning Operator
    • None
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Tuned profile is failing to remove kernel arguments in ocp 4.14.25

      Steps to Reproduce:

      • create and apply tuned-custom.yaml
      # cat 
      apiVersion: tuned.openshift.io/v1
      kind: Tuned
      metadata:
        name: custom-tuned
        namespace: openshift-cluster-node-tuning-operator
      spec:
        profile:
        - name: custom-tuned
          data: |
            [main]
            summary=Custom profile to remove slub_debug kernel parameter
      
            [bootloader]
            cmdline_custom_tuned=-page_poison=1
        recommend:
        - priority: 20
          machineConfigLabels:
            machineconfiguration.openshift.io/role: worker
          profile: custom-tuned
          operand:
            debug: true
      
      # oc create -f tuned-custom.yaml
      
      # # oc get tuned
      NAME           AGE
      custom-tuned   11m
      default        22d
      rendered       22d
      
      # oc get profile
      NAME                               TUNED                     APPLIED   DEGRADED   AGE
      mno-ctlplane-0.5g-deployment.lab   openshift-control-plane   True      False      22d
      mno-ctlplane-1.5g-deployment.lab   openshift-control-plane   True      False      22d
      mno-ctlplane-2.5g-deployment.lab   openshift-control-plane   True      False      22d
      mno-worker-0.5g-deployment.lab     custom-tuned              True      False      22d
      mno-worker-1.5g-deployment.lab     custom-tuned              True      False      22d
      mno-worker-2.5g-deployment.lab     custom-tuned              True      False      22d
      
      • check tuned pod logs and got an error "Executing 'rpm-ostree status' error: [Errno 2] No such file or directory: 'rpm-ostree'"
      # oc logs cluster-node-tuning-operator-85b475b4c6-grjtg
      ...
      I0704 17:46:29.866630       1 controller.go:552] updated Tuned rendered
      I0704 17:46:29.888531       1 controller.go:766] updated profile mno-worker-1.5g-deployment.lab [custom-tuned]
      I0704 17:46:29.912993       1 controller.go:766] updated profile mno-worker-2.5g-deployment.lab [custom-tuned]
      I0704 17:46:29.941904       1 controller.go:766] updated profile mno-worker-0.5g-deployment.lab [custom-tuned]
      ...
      
      # oc logs tuned-c6snh -f
      ...
      I0704 17:47:36.926653    3402 controller.go:848] active profile (openshift-node) != recommended profile (custom-tuned)
      2024-07-04 17:47:36,927 INFO     tuned.daemon.controller: terminating controller
      2024-07-04 17:47:36,928 INFO     tuned.daemon.daemon: stopping tuning
      2024-07-04 17:47:36,954 INFO     tuned.daemon.daemon: terminating TuneD and not rolling back any changes due to 'rollback' option in '/etc/tuned/tuned-main.conf'
      I0704 17:47:36.990860    3402 controller.go:573] starting tuned...
      2024-07-04 17:47:37,111 INFO     tuned.daemon.application: TuneD: 2.22.1, kernel: 5.14.0-284.66.1.el9_2.x86_64
      2024-07-04 17:47:37,111 INFO     tuned.daemon.application: dynamic tuning is globally disabled
      2024-07-04 17:47:37,116 DEBUG    tuned.daemon.daemon: initializing daemon
      2024-07-04 17:47:37,116 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)
      2024-07-04 17:47:37,116 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.
      2024-07-04 17:47:37,117 INFO     tuned.daemon.daemon: Using 'custom-tuned' profile
      2024-07-04 17:47:37,118 INFO     tuned.profiles.loader: loading profile: custom-tuned
      2024-07-04 17:47:37,119 INFO     tuned.daemon.controller: starting controller
      2024-07-04 17:47:37,119 INFO     tuned.daemon.daemon: starting tuning
      2024-07-04 17:47:37,119 DEBUG    tuned.plugins.repository: creating plugin bootloader
      2024-07-04 17:47:37,119 DEBUG    tuned.utils.plugin_loader: loading module tuned.plugins.plugin_bootloader
      2024-07-04 17:47:37,121 DEBUG    tuned.units.manager: creating 'bootloader' (bootloader)
      2024-07-04 17:47:37,121 DEBUG    tuned.plugins.base: initializing instance bootloader (bootloader)
      2024-07-04 17:47:37,121 DEBUG    tuned.utils.commands: Read data from file: '/etc/default/grub' > ''
      2024-07-04 17:47:37,121 INFO     tuned.plugins.plugin_bootloader: cannot read '/etc/default/grub'
      2024-07-04 17:47:37,122 DEBUG    tuned.utils.commands: Executing ['rpm-ostree', 'status'].
      2024-07-04 17:47:37,124 DEBUG    tuned.plugins.plugin_bootloader: rpm-ostree status output stdout:
      
      stderr:
      Executing 'rpm-ostree status' error: [Errno 2] No such file or directory: 'rpm-ostree'
      2024-07-04 17:47:37,124 DEBUG    tuned.hardware.inventory: starting monitor observer
      2024-07-04 17:47:37,125 INFO     tuned.daemon.daemon: static tuning from profile 'custom-tuned' applied
      I0704 17:47:37.149546    3402 controller.go:494] extracting cloud provider name to /var/lib/tuned/provider
      I0704 17:47:37.149570    3402 controller.go:1056] updated Profile mno-worker-0.5g-deployment.lab bootcmdline: 
      I0704 17:47:37.149860    3402 controller.go:536] written "/etc/tuned/recommend.d/50-openshift.conf" to set TuneD profile custom-tuned
      I0704 17:47:37.150055    3402 controller.go:852] active and recommended profile (custom-tuned) match; profile change will not trigger profile reload
      

      as per the doc hereĀ https://github.com/redhat-performance/tuned/blob/master/tuned/plugins/plugin_bootloader.py#L63 the bootloader used seems correct

       It is also possible to remove kernel options by the following syntax:
      	+
      	[subs="+quotes,+macros"]
      	----
      	cmdline__suffix__=-__arg1__ __arg2__ ... __argN__
      	----
      	+
      Such kernel options will not be concatenated and thus removed during 	the final kernel command line construction.
      	...
      	An example of a custom profile `my_profile` that removes the
      	[option]`rhgb quiet` options from the kernel command line (if
      	previously added by *TuneD*):
      	
      	----
      	[bootloader]
      	cmdline_my_profile=-rhgb quiet
      	----
      	====
      

      Actual results:

      Tuned configuration is not applied

      Expected results:

      Tuned configuration being applied

              team-nto Team NTO
              rhn-support-jclaretm Jorge Claret Membrado
              Liquan Cui Liquan Cui
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: