Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-11342

tuned.utils.commands: Writing to file '/sys/block/dm-2/queue/read_ahead_kb' error: '[Errno 2] No such file or directory: '/sys/block/dm-2/queue/read_ahead_kb''

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Major Major
    • rhel-9.4
    • rhel-9.2.0.z
    • tuned
    • None
    • tuned-2.22.0-0.1.rc1.el9
    • None
    • Important
    • Patch, Upstream
    • rhel-sst-cs-net-perf-services
    • ssg_core_services
    • 26
    • None
    • Hide

      11/5: This is currently targeted to  FDP 23.j
      10/30: Yellow. No response yet from RHEL Eng in Jira. No fix identified so unable to forecast at this time.

      Show
      11/5: This is currently targeted to  FDP 23.j 10/30: Yellow. No response yet from RHEL Eng in Jira. No fix identified so unable to forecast at this time.
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None

      Description of problem:

      [root@master0 ~]# oc get performanceprofiles.performance.openshift.io
      NAME             AGE
      master-profile   94m
      [root@master0 ~]# oc get performanceprofiles.performance.openshift.io master-profile
      NAME             AGE
      master-profile   94m
      [root@master0 ~]# oc describe performanceprofiles.performance.openshift.io master-profile
      Name:         master-profile
      Namespace:
      Labels:       <none>
      Annotations:  kubeletconfig.experimental:
                      {"allowedUnsafeSysctls":["net.core.somaxconn","kernel.msg*","net.ipv6.conf.all.accept_ra","net.ipv6.conf.default.accept_ra","net.ipv6.conf...
      API Version:  performance.openshift.io/v2
      Kind:         PerformanceProfile
      Metadata:
        Creation Timestamp:  2023-10-02T13:18:55Z
        Finalizers:
          foreground-deletion
        Generation:        1
        Resource Version:  17649
        UID:               d6cfda76-f320-4df8-83af-3f09e10965ac
      Spec:
        Additional Kernel Args:
          nmi_watchdog=0
          audit=1
          mce=off
          processor.max_cstate=1
          intel_idle.max_cstate=0
          rcupdate.rcu_normal_after_boot=0
          iommu=pt
          intel_iommu=on
        Cpu:
          Isolated:                           2-31,34-63
          Reserved:                           0-1,32-33
        Globally Disable Irq Load Balancing:  false
        Hugepages:
          Default Hugepages Size:  1G
          Pages:
            Count:  21
            Size:   1G
        Machine Config Pool Selector:
          pools.operator.machineconfiguration.openshift.io/master:
        Node Selector:
          node-role.kubernetes.io/master:
        Numa:
          Topology Policy:  best-effort
        Real Time Kernel:
          Enabled:  false
      Status:
        Conditions:
          Last Heartbeat Time:   2023-10-02T13:18:56Z
          Last Transition Time:  2023-10-02T13:18:56Z
          Status:                True
          Type:                  Available
          Last Heartbeat Time:   2023-10-02T13:18:56Z
          Last Transition Time:  2023-10-02T13:18:56Z
          Status:                True
          Type:                  Upgradeable
          Last Heartbeat Time:   2023-10-02T13:18:56Z
          Last Transition Time:  2023-10-02T13:18:56Z
          Status:                False
          Type:                  Progressing
          Last Heartbeat Time:   2023-10-02T13:18:56Z
          Last Transition Time:  2023-10-02T13:18:56Z
          Status:                False
          Type:                  Degraded
        Runtime Class:           performance-master-profile
        Tuned:                   openshift-cluster-node-tuning-operator/openshift-node-performance-master-profile
      Events:
        Type    Reason              Age                 From                            Message
        ----    ------              ----                ----                            -------
        Normal  Creation succeeded  91m (x16 over 94m)  performance-profile-controller  Succeeded to create all components
        Normal  Creation succeeded  66m (x7 over 80m)   performance-profile-controller  Succeeded to create all components
      
      [root@master0 ~]# oc get performanceprofiles.performance.openshift.io master-profile -o yaml
      apiVersion: performance.openshift.io/v2
      kind: PerformanceProfile
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"performance.openshift.io/v2","kind":"PerformanceProfile","metadata":{"annotations":{"kubeletconfig.experimental":"{\"allowedUnsafeSysctls\":[\"net.core.somaxconn\",\"kernel.msg*\",\"net.ipv6.conf.all.accept_ra\",\"net.ipv6.conf.default.accept_ra                                                                                                                                                                \",\"net.ipv6.conf.all.disable_ipv6\",\"net.ipv4.conf.default.disable_policy\",\"net.ipv4.conf.all.disable_policy\",\"net.ipv6.conf.default.disable_policy\",\"net.ipv6.conf.all.disable_policy\",\"net.ipv4.conf.default.rp_filter\",\"net.ipv4.conf.all.rp_filter\"],\"s                                                                                                                                                                ystemReserved\":{\"memory\": \"11.78Gi\"},\"kubeReserved\":{\"memory\": \"500Mi\"},\"tlsCipherSuites\":[\"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\",\"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\",\"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\",\"TLS_ECDHE_ECDSA_WITH_AES_128_GCM                                                                                                                                                                _SHA256\"],\"eventRecordQPS\":5,\"imageGCHighThresholdPercent\":80,\"imageGCLowThresholdPercent\":75,\"imageMinimumGCAge\":\"60m\",\"makeIPTablesUtilChains\":true,\"protectKernelDefaults\":true,\"streamingConnectionIdleTimeout\":\"4h0m0s\",\"evictionSoft\":{\"memory                                                                                                                                                                .available\":\"500Mi\",\"nodefs.available\":\"10%\",\"nodefs.inodesFree\":\"5%\",\"imagefs.available\":\"15%\",\"imagefs.inodesFree\":\"10%\"},\"evictionSoftGracePeriod\":{\"memory.available\":\"1m30s\",\"nodefs.available\":\"1m30s\",\"nodefs.inodesFree\":\"1m30s\",                                                                                                                                                                \"imagefs.available\":\"1m30s\",\"imagefs.inodesFree\":\"1m30s\"},\"evictionHard\":{\"memory.available\":\"200Mi\",\"nodefs.available\":\"5%\",\"nodefs.inodesFree\":\"4%\",\"imagefs.available\":\"10%\",\"imagefs.inodesFree\":\"5%\"}}\n"},"name":"master-profile"},"sp                                                                                                                                                                ec":{"additionalKernelArgs":["nmi_watchdog=0","audit=1","mce=off","processor.max_cstate=1","intel_idle.max_cstate=0","rcupdate.rcu_normal_after_boot=0","iommu=pt","intel_iommu=on"],"cpu":{"isolated":"2-31,34-63","reserved":"0-1,32-33"},"globallyDisableIrqLoadBalanci                                                                                                                                                                ng":false,"hugepages":{"defaultHugepagesSize":"1G","pages":[{"count":21,"size":"1G"}]},"machineConfigPoolSelector":{"pools.operator.machineconfiguration.openshift.io/master":""},"nodeSelector":{"node-role.kubernetes.io/master":""},"numa":{"topologyPolicy":"best-effo                                                                                                                                                                rt"},"realTimeKernel":{"enabled":false}}}
          kubeletconfig.experimental: |
            {"allowedUnsafeSysctls":["net.core.somaxconn","kernel.msg*","net.ipv6.conf.all.accept_ra","net.ipv6.conf.default.accept_ra","net.ipv6.conf.all.disable_ipv6","net.ipv4.conf.default.disable_policy","net.ipv4.conf.all.disable_policy","net.ipv6.conf.default.disabl                                                                                                                                                                e_policy","net.ipv6.conf.all.disable_policy","net.ipv4.conf.default.rp_filter","net.ipv4.conf.all.rp_filter"],"systemReserved":{"memory": "11.78Gi"},"kubeReserved":{"memory": "500Mi"},"tlsCipherSuites":["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_A                                                                                                                                                                ES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"],"eventRecordQPS":5,"imageGCHighThresholdPercent":80,"imageGCLowThresholdPercent":75,"imageMinimumGCAge":"60m","makeIPTablesUtilChains":true,"protectKernelDefaults":                                                                                                                                                                true,"streamingConnectionIdleTimeout":"4h0m0s","evictionSoft":{"memory.available":"500Mi","nodefs.available":"10%","nodefs.inodesFree":"5%","imagefs.available":"15%","imagefs.inodesFree":"10%"},"evictionSoftGracePeriod":{"memory.available":"1m30s","nodefs.available"                                                                                                                                                                :"1m30s","nodefs.inodesFree":"1m30s","imagefs.available":"1m30s","imagefs.inodesFree":"1m30s"},"evictionHard":{"memory.available":"200Mi","nodefs.available":"5%","nodefs.inodesFree":"4%","imagefs.available":"10%","imagefs.inodesFree":"5%"}}
        creationTimestamp: "2023-10-02T13:18:55Z"
        finalizers:
        - foreground-deletion
        generation: 1
        name: master-profile
        resourceVersion: "17649"
        uid: d6cfda76-f320-4df8-83af-3f09e10965ac
      spec:
        additionalKernelArgs:
        - nmi_watchdog=0
        - audit=1
        - mce=off
        - processor.max_cstate=1
        - intel_idle.max_cstate=0
        - rcupdate.rcu_normal_after_boot=0
        - iommu=pt
        - intel_iommu=on
        cpu:
          isolated: 2-31,34-63
          reserved: 0-1,32-33
        globallyDisableIrqLoadBalancing: false
        hugepages:
          defaultHugepagesSize: 1G
          pages:
          - count: 21
            size: 1G
        machineConfigPoolSelector:
          pools.operator.machineconfiguration.openshift.io/master: ""
        nodeSelector:
          node-role.kubernetes.io/master: ""
        numa:
          topologyPolicy: best-effort
        realTimeKernel:
          enabled: false
      status:
        conditions:
        - lastHeartbeatTime: "2023-10-02T13:18:56Z"
          lastTransitionTime: "2023-10-02T13:18:56Z"
          status: "True"
          type: Available
        - lastHeartbeatTime: "2023-10-02T13:18:56Z"
          lastTransitionTime: "2023-10-02T13:18:56Z"
          status: "True"
          type: Upgradeable
        - lastHeartbeatTime: "2023-10-02T13:18:56Z"
          lastTransitionTime: "2023-10-02T13:18:56Z"
          status: "False"
          type: Progressing
        - lastHeartbeatTime: "2023-10-02T13:18:56Z"
          lastTransitionTime: "2023-10-02T13:18:56Z"
          status: "False"
          type: Degraded
        runtimeClass: performance-master-profile
        tuned: openshift-cluster-node-tuning-operator/openshift-node-performance-master-profile [root@master0 ~]# oc get pods -n openshift-cluster-node-tuning-operator
      NAME                                           READY   STATUS    RESTARTS   AGE
      cluster-node-tuning-operator-f94c5888d-5c2mg   1/1     Running   1          110m
      tuned-fv66v                                    1/1     Running   1          108m
      [root@master0 ~]# oc logs -n openshift-cluster-node-tuning-operator cluster-node-tuning-operator-f94c5888d-5c2mg
      I1002 13:27:50.015282       1 main.go:75] Go Version: go1.20.5 X:strictfipsruntime
      I1002 13:27:50.015369       1 main.go:76] Go OS/Arch: linux/amd64
      I1002 13:27:50.015374       1 main.go:77] node-tuning Version: v4.14.0-202309131203.p0.g500b852.assembly.stream-0-g375c9c1-dirty
      I1002 13:27:50.015600       1 leaderelection.go:122] The leader election gives 4 retries and allows for 30s of clock skew. The kube-apiserver downtime tolerance is 78s. Worst non-graceful lease acquisition is 2m43s. Worst graceful lease acquisition is {26s}.
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"performance.openshift.io/v1, Kind=PerformanceProfile"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.builder","msg":"skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called","GVK":"performance.openshift.io/v1, Kind=PerformanceProfile"                                                                                                                                                                }
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/convert"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"performance.openshift.io/v1, Kind=PerformanceProfile"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.builder","msg":"skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","GVK":"performance.openshift.io/v2, Kind=PerformanceProfile"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.builder","msg":"Registering a validating webhook","GVK":"performance.openshift.io/v2, Kind=PerformanceProfile","path":"/validate-performance-openshift-io-v2-performanceprofile"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.webhook","msg":"Registering webhook","path":"/validate-performance-openshift-io-v2-performanceprofile"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.builder","msg":"Conversion webhook enabled","GVK":"performance.openshift.io/v2, Kind=PerformanceProfile"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.webhook.webhooks","msg":"Starting webhook server"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","msg":"starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
      I1002 13:27:50.066167       1 leaderelection.go:245] attempting to acquire leader lease openshift-cluster-node-tuning-operator/node-tuning-operator-lock...
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":4343}
      {"level":"info","ts":"2023-10-02T13:27:50Z","logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
      I1002 13:32:47.131167       1 leaderelection.go:255] successfully acquired lease openshift-cluster-node-tuning-operator/node-tuning-operator-lock
      I1002 13:32:47.131314       1 controller.go:1308] starting Tuned controller
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting EventSource","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile","source":"kind source: *v2.PerformanceProfile"}
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting EventSource","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile","source":"kind source: *v1.MachineConfig"}
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting EventSource","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile","source":"kind source: *v1.KubeletConfig"}
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting EventSource","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile","source":"kind source: *v1.Tuned"}
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting EventSource","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile","source":"kind source: *v1.RuntimeClass"}
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting EventSource","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile","source":"kind source: *v1.MachineConfigPool"}
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting EventSource","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile","source":"kind source: *v1.Profile"}
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting EventSource","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile","source":"kind source: *v1.ContainerRuntimeConfig"}
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting Controller","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile"}
      I1002 13:32:47.532089       1 controller.go:1408] started events processor/controller
      I1002 13:32:47.538480       1 server.go:102] starting metrics server
      I1002 13:32:47.545830       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:32:47.550290       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:32:47.552513       1 status.go:303] 1/1 Profiles failed to be applied
      {"level":"info","ts":"2023-10-02T13:32:47Z","msg":"Starting workers","controller":"performanceprofile","controllerGroup":"performance.openshift.io","controllerKind":"PerformanceProfile","worker count":1}
      I1002 13:32:47.697282       1 performanceprofile_controller.go:389] Reconciling PerformanceProfile
      I1002 13:32:47.705943       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:32:47.898005       1 performanceprofile_controller.go:498] using "runc" as high-performance runtime class container-runtime for profile "master-profile"
      I1002 13:32:47.900073       1 resources.go:93] Update machine-config "50-performance-master-profile"
      I1002 13:32:47.915818       1 performanceprofile_controller.go:389] Reconciling PerformanceProfile
      I1002 13:32:47.916098       1 performanceprofile_controller.go:498] using "runc" as high-performance runtime class container-runtime for profile "master-profile"
      I1002 13:32:47.917265       1 resources.go:93] Update machine-config "50-performance-master-profile"
      I1002 13:33:16.196977       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:34:25.709739       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:34:26.095201       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:34:28.455080       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:34:30.723888       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:34:32.449348       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:34:37.937569       1 performanceprofile_controller.go:389] Reconciling PerformanceProfile
      I1002 13:34:37.937818       1 performanceprofile_controller.go:498] using "runc" as high-performance runtime class container-runtime for profile "master-profile"
      I1002 13:34:37.939225       1 resources.go:93] Update machine-config "50-performance-master-profile"
      I1002 13:36:48.578298       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:36:49.496495       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:36:51.052850       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:36:56.367802       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:37:04.451209       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:37:04.690937       1 performanceprofile_controller.go:389] Reconciling PerformanceProfile
      I1002 13:37:04.691267       1 performanceprofile_controller.go:498] using "runc" as high-performance runtime class container-runtime for profile "master-profile"
      I1002 13:37:04.692320       1 resources.go:93] Update machine-config "50-performance-master-profile"
      I1002 13:44:43.498764       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:44:43.652885       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:44:48.530724       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:44:48.850622       1 performanceprofile_controller.go:389] Reconciling PerformanceProfile
      I1002 13:44:48.850935       1 performanceprofile_controller.go:498] using "runc" as high-performance runtime class container-runtime for profile "master-profile"
      I1002 13:44:48.852410       1 resources.go:93] Update machine-config "50-performance-master-profile"
      I1002 13:44:48.863134       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:44:54.493174       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:45:16.119542       1 performanceprofile_controller.go:389] Reconciling PerformanceProfile
      I1002 13:45:16.119683       1 performanceprofile_controller.go:498] using "runc" as high-performance runtime class container-runtime for profile "master-profile"
      I1002 13:45:16.120812       1 resources.go:93] Update machine-config "50-performance-master-profile"
      I1002 13:45:16.129461       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:46:06.280294       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:46:07.054496       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:46:08.406523       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:46:09.633907       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:46:12.390753       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:46:19.154154       1 performanceprofile_controller.go:389] Reconciling PerformanceProfile
      I1002 13:46:19.154341       1 performanceprofile_controller.go:498] using "runc" as high-performance runtime class container-runtime for profile "master-profile"
      I1002 13:46:19.155544       1 resources.go:93] Update machine-config "50-performance-master-profile"
      I1002 13:56:06.281430       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:56:06.321260       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:56:07.049570       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:56:08.392771       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:56:09.635682       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 13:56:12.390812       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:06:06.281381       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:06:07.051054       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:06:08.395035       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:06:09.635747       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:06:12.391629       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:16:06.282676       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:16:07.054294       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:16:08.394744       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:16:09.637315       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:16:12.391603       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:26:06.282951       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:26:07.051746       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:26:08.394960       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:26:09.637878       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:26:12.392653       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:36:06.283610       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:36:07.051935       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:36:08.396003       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:36:09.639539       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:36:12.393670       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:46:06.283905       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:46:07.052005       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:46:08.396303       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:46:09.640008       1 status.go:303] 1/1 Profiles failed to be applied
      I1002 14:46:12.393930       1 status.go:303] 1/1 Profiles failed to be applied [root@master0 ~]# oc logs -n openshift-cluster-node-tuning-operator tuned-fv66v
      I1002 13:26:59.906215   13595 controller.go:1306] starting openshift-tuned v4.14.0-202309131203.p0.g500b852.assembly.stream-0-g375c9c1-dirty
      I1002 13:27:15.141988   13595 trace.go:236] Trace[993308075]: "Reflector ListAndWatch" name:github.com/openshift/cluster-node-tuning-operator/pkg/generated/informers/externalversions/factory.go:116 (02-Oct-2023 13:26:59.932) (total time: 15209ms):
      Trace[993308075]: ---"Objects listed" error:<nil> 15209ms (13:27:15.141)
      Trace[993308075]: [15.20915869s] [15.20915869s] END
      I1002 13:27:15.142581   13595 trace.go:236] Trace[926523190]: "Reflector ListAndWatch" name:github.com/openshift/cluster-node-tuning-operator/pkg/generated/informers/externalversions/factory.go:116 (02-Oct-2023 13:26:59.932) (total time: 15209ms):
      Trace[926523190]: ---"Objects listed" error:<nil> 15209ms (13:27:15.142)
      Trace[926523190]: [15.209793488s] [15.209793488s] END
      I1002 13:27:15.142788   13595 controller.go:363] disabling system tuned...
      I1002 13:27:15.232203   13595 controller.go:1155] started events processors
      I1002 13:27:15.232290   13595 controller.go:1172] started controller
      I1002 13:27:15.232304   13595 controller.go:393] extracting TuneD profiles
      I1002 13:27:15.233288   13595 controller.go:482] extracting cloud provider name to /var/lib/tuned/provider
      I1002 13:27:15.233478   13595 controller.go:524] written "/etc/tuned/recommend.d/50-openshift.conf" to set TuneD profile performance-patch-master
      I1002 13:27:15.233548   13595 controller.go:561] starting tuned...
      2023-10-02 13:27:15,431 INFO     tuned.daemon.application: TuneD: 2.20.0, kernel: 5.14.0-284.32.1.el9_2.x86_64
      2023-10-02 13:27:15,431 INFO     tuned.daemon.application: dynamic tuning is globally disabled
      2023-10-02 13:27:15,441 INFO     tuned.daemon.daemon: using sleep interval of 1 second(s)
      2023-10-02 13:27:15,441 INFO     tuned.daemon.daemon: Running in automatic mode, checking what profile is recommended for your configuration.
      2023-10-02 13:27:15,442 INFO     tuned.daemon.daemon: Using 'performance-patch-master' profile
      2023-10-02 13:27:15,443 INFO     tuned.profiles.loader: loading profile: performance-patch-master
      2023-10-02 13:27:15,557 INFO     tuned.daemon.controller: starting controller
      2023-10-02 13:27:15,558 INFO     tuned.daemon.daemon: starting tuning
      2023-10-02 13:27:15,621 INFO     tuned.plugins.base: instance cpu: assigning devices cpu57, cpu61, cpu26, cpu12, cpu28, cpu18, cpu63, cpu27, cpu4, cpu56, cpu42, cpu47, cpu59, cpu10, cpu36, cpu58, cpu38, cpu13, cpu7, cpu33, cpu0, cpu51, cpu24, cpu21, cpu35, cpu2, cpu25, cpu39, cpu15, cpu16, cpu20, cpu32, cpu44, cpu5, cpu8, cpu31, cpu62, cpu22, cpu9, cpu50, cpu6, cpu11, cpu40, cpu29, cpu46, cpu23, cpu49, cpu30, cpu60, cpu55, cpu54, cpu48, cpu14, cpu41, cpu1, cpu43, cpu17, cpu3, cpu45, cpu52, cpu53, cpu19, cpu34, cpu37
      2023-10-02 13:27:15,627 INFO     tuned.plugins.plugin_cpu: We are running on an x86 GenuineIntel platform
      2023-10-02 13:27:15,642 INFO     tuned.plugins.plugin_disk: Device 'nvme0n1' not supported by hdparm
      2023-10-02 13:27:15,644 INFO     tuned.plugins.plugin_disk: Device 'dm-0' not supported by hdparm
      2023-10-02 13:27:15,647 INFO     tuned.plugins.plugin_disk: Device 'dm-1' not supported by hdparm
      2023-10-02 13:27:15,649 INFO     tuned.plugins.plugin_disk: Device 'dm-2' not supported by hdparm
      2023-10-02 13:27:15,650 INFO     tuned.plugins.base: instance disk: assigning devices nvme0n1, dm-1, dm-2, sda, dm-0
      2023-10-02 13:27:15,653 INFO     tuned.plugins.base: instance net: assigning devices enp1s0, ens43f0, ens43f2, ens43f3, ens43f1, usb0
      2023-10-02 13:27:15,840 INFO     tuned.plugins.plugin_rtentsk: opened SOF_TIMESTAMPING_OPT_TX_SWHW socket
      2023-10-02 13:27:15,840 INFO     tuned.plugins.plugin_bootloader: cannot read '/etc/default/grub'
      2023-10-02 13:27:15,843 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu26'
      2023-10-02 13:27:15,843 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu18'
      2023-10-02 13:27:15,843 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu42'
      2023-10-02 13:27:15,843 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu10'
      2023-10-02 13:27:15,844 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu38'
      2023-10-02 13:27:15,844 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu13'
      2023-10-02 13:27:15,844 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu0'
      2023-10-02 13:27:15,844 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu51'
      2023-10-02 13:27:15,844 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu39'
      2023-10-02 13:27:15,844 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu35'
      2023-10-02 13:27:15,844 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu44'
      2023-10-02 13:27:15,845 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu49'
      2023-10-02 13:27:15,845 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu32'
      2023-10-02 13:27:15,845 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu5'
      2023-10-02 13:27:15,845 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu31'
      2023-10-02 13:27:15,845 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu22'
      2023-10-02 13:27:15,845 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu6'
      2023-10-02 13:27:15,846 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu11'
      2023-10-02 13:27:15,846 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu23'
      2023-10-02 13:27:15,846 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu30'
      2023-10-02 13:27:15,846 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu60'
      2023-10-02 13:27:15,846 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu55'
      2023-10-02 13:27:15,846 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu48'
      2023-10-02 13:27:15,847 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu41'
      2023-10-02 13:27:15,847 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu1'
      2023-10-02 13:27:15,847 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu17'
      2023-10-02 13:27:15,847 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu53'
      2023-10-02 13:27:15,847 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu37'
      2023-10-02 13:27:15,847 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu57'
      2023-10-02 13:27:15,847 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu61'
      2023-10-02 13:27:15,848 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu12'
      2023-10-02 13:27:15,848 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu28'
      2023-10-02 13:27:15,848 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu63'
      2023-10-02 13:27:15,848 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu27'
      2023-10-02 13:27:15,848 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu4'
      2023-10-02 13:27:15,848 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu56'
      2023-10-02 13:27:15,849 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu47'
      2023-10-02 13:27:15,849 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu59'
      2023-10-02 13:27:15,849 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu36'
      2023-10-02 13:27:15,849 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu58'
      2023-10-02 13:27:15,849 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu7'
      2023-10-02 13:27:15,849 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu33'
      2023-10-02 13:27:15,850 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu24'
      2023-10-02 13:27:15,850 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu21'
      2023-10-02 13:27:15,850 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu2'
      2023-10-02 13:27:15,850 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu25'
      2023-10-02 13:27:15,850 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu15'
      2023-10-02 13:27:15,850 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu16'
      2023-10-02 13:27:15,851 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu20'
      2023-10-02 13:27:15,851 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu8'
      2023-10-02 13:27:15,851 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu62'
      2023-10-02 13:27:15,851 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu9'
      2023-10-02 13:27:15,851 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu50'
      2023-10-02 13:27:15,852 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu40'
      2023-10-02 13:27:15,852 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu29'
      2023-10-02 13:27:15,852 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu46'
      2023-10-02 13:27:15,852 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu54'
      2023-10-02 13:27:15,852 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu14'
      2023-10-02 13:27:15,852 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu43'
      2023-10-02 13:27:15,853 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu3'
      2023-10-02 13:27:15,853 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu45'
      2023-10-02 13:27:15,853 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu52'
      2023-10-02 13:27:15,853 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu19'
      2023-10-02 13:27:15,853 INFO     tuned.plugins.plugin_cpu: setting governor 'performance' on cpu 'cpu34'
      2023-10-02 13:27:15,863 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu26'
      2023-10-02 13:27:15,868 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu18'
      2023-10-02 13:27:15,873 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu42'
      2023-10-02 13:27:15,878 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu10'
      2023-10-02 13:27:15,882 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu38'
      2023-10-02 13:27:15,887 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu13'
      2023-10-02 13:27:15,892 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu0'
      2023-10-02 13:27:15,897 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu51'
      2023-10-02 13:27:15,902 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu39'
      2023-10-02 13:27:15,907 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu35'
      2023-10-02 13:27:15,912 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu44'
      2023-10-02 13:27:15,916 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu49'
      2023-10-02 13:27:15,921 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu32'
      2023-10-02 13:27:15,926 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu5'
      2023-10-02 13:27:15,931 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu31'
      2023-10-02 13:27:15,936 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu22'
      2023-10-02 13:27:15,941 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu6'
      2023-10-02 13:27:15,946 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu11'
      2023-10-02 13:27:15,951 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu23'
      2023-10-02 13:27:15,955 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu30'
      2023-10-02 13:27:15,960 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu60'
      2023-10-02 13:27:15,965 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu55'
      2023-10-02 13:27:15,970 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu48'
      2023-10-02 13:27:15,975 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu41'
      2023-10-02 13:27:15,980 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu1'
      2023-10-02 13:27:15,985 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu17'
      2023-10-02 13:27:15,989 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu53'
      2023-10-02 13:27:15,994 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu37'
      2023-10-02 13:27:15,999 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu57'
      2023-10-02 13:27:16,004 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu61'
      2023-10-02 13:27:16,010 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu12'
      2023-10-02 13:27:16,015 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu28'
      2023-10-02 13:27:16,022 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu63'
      2023-10-02 13:27:16,029 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu27'
      2023-10-02 13:27:16,035 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu4'
      2023-10-02 13:27:16,041 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu56'
      2023-10-02 13:27:16,046 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu47'
      2023-10-02 13:27:16,052 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu59'
      2023-10-02 13:27:16,057 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu36'
      2023-10-02 13:27:16,062 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu58'
      2023-10-02 13:27:16,066 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu7'
      2023-10-02 13:27:16,071 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu33'
      2023-10-02 13:27:16,076 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu24'
      2023-10-02 13:27:16,081 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu21'
      2023-10-02 13:27:16,086 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu2'
      2023-10-02 13:27:16,091 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu25'
      2023-10-02 13:27:16,096 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu15'
      2023-10-02 13:27:16,101 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu16'
      2023-10-02 13:27:16,105 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu20'
      2023-10-02 13:27:16,110 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu8'
      2023-10-02 13:27:16,115 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu62'
      2023-10-02 13:27:16,120 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu9'
      2023-10-02 13:27:16,125 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu50'
      2023-10-02 13:27:16,130 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu40'
      2023-10-02 13:27:16,135 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu29'
      2023-10-02 13:27:16,140 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu46'
      2023-10-02 13:27:16,145 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu54'
      2023-10-02 13:27:16,150 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu14'
      2023-10-02 13:27:16,155 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu43'
      2023-10-02 13:27:16,160 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu3'
      2023-10-02 13:27:16,165 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu45'
      2023-10-02 13:27:16,170 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu52'
      2023-10-02 13:27:16,177 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu19'
      2023-10-02 13:27:16,185 INFO     tuned.plugins.plugin_cpu: energy_perf_bias successfully set to 'performance' on cpu 'cpu34'
      2023-10-02 13:27:16,186 INFO     tuned.plugins.plugin_cpu: setting new cpu latency 1
      2023-10-02 13:27:16,190 INFO     tuned.plugins.plugin_sysctl: reapplying system sysctl
      2023-10-02 13:27:16,191 INFO     tuned.plugins.plugin_sysctl: Overriding sysctl parameter 'net.core.rps_default_mask' from '${not_isolated_cpumask}' to '00000003,00000003'
      2023-10-02 13:27:16,625 INFO     tuned.plugins.plugin_systemd: setting 'CPUAffinity' to '0 1 32 33' in the '/etc/systemd/system.conf'
      2023-10-02 13:27:17,147 INFO     tuned.plugins.plugin_script: calling script '/usr/lib/tuned/cpu-partitioning/script.sh' with arguments '['start']'
      2023-10-02 13:27:17,296 ERROR    tuned.plugins.plugin_script: script '/usr/lib/tuned/cpu-partitioning/script.sh' returned error code: 1
      2023-10-02 13:27:17,298 INFO     tuned.plugins.plugin_bootloader: installing additional boot command line parameters to grub2
      2023-10-02 13:27:17,298 INFO     tuned.plugins.plugin_bootloader: cannot find grub.cfg to patch
      E1002 13:27:17.299651   13595 controller.go:899] unable to sync(daemon/) requeued (0)
      E1002 13:27:17.299672   13595 controller.go:899] unable to sync(daemon/) requeued (1)
      2023-10-02 13:27:17,299 INFO     tuned.daemon.daemon: static tuning from profile 'performance-patch-master' applied
      2023-10-02 13:44:27,638 INFO     tuned.plugins.hotplug: device 'dm-2' removed
      2023-10-02 13:44:27,639 ERROR    tuned.utils.commands: Writing to file '/sys/block/dm-2/queue/read_ahead_kb' error: '[Errno 2] No such file or directory: '/sys/block/dm-2/queue/read_ahead_kb''
      2023-10-02 13:44:27,663 INFO     tuned.plugins.hotplug: device 'dm-2' removed
      2023-10-02 13:44:27,851 INFO     tuned.plugins.hotplug: device 'dm-1' removed
      2023-10-02 13:44:27,855 ERROR    tuned.utils.commands: Writing to file '/sys/block/dm-1/queue/read_ahead_kb' error: '[Errno 2] No such file or directory: '/sys/block/dm-1/queue/read_ahead_kb''
      2023-10-02 13:44:27,957 INFO     tuned.plugins.hotplug: device 'dm-1' removed
      2023-10-02 13:44:28,003 INFO     tuned.plugins.hotplug: device 'dm-0' removed
      2023-10-02 13:44:28,006 INFO     tuned.plugins.hotplug: device 'dm-0' removed
      2023-10-02 13:44:28,008 INFO     tuned.plugins.hotplug: device 'dm-0' added
      2023-10-02 13:44:28,008 INFO     tuned.plugins.hotplug: instance disk: adding new device dm-0
      2023-10-02 13:44:28,019 INFO     tuned.plugins.hotplug: device 'dm-1' added
      2023-10-02 13:44:28,019 INFO     tuned.plugins.hotplug: instance disk: adding new device dm-1
      2023-10-02 13:44:28,464 INFO     tuned.plugins.hotplug: device 'dm-2' added
      2023-10-02 13:44:28,465 INFO     tuned.plugins.hotplug: instance disk: adding new device dm-2
      2023-10-02 13:44:28,477 INFO     tuned.plugins.hotplug: device 'dm-3' added
      2023-10-02 13:44:28,477 INFO     tuned.plugins.hotplug: instance disk: adding new device dm-3
      2023-10-02 13:44:28,533 INFO     tuned.plugins.hotplug: device 'dm-4' added
      2023-10-02 13:44:28,533 INFO     tuned.plugins.hotplug: instance disk: adding new device dm-4
      

      Version-Release number of selected component (if applicable):

      4.14.0-rc.1

      How reproducible:

      apply the PAO config on the OCP cluster and check the tuned-operator logs

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

      I1002 13:32:47.545830       1 status.go:303] 1/1 Profiles failed to be applied
      

      Expected results:

       

      Additional info:

      sosreport and must-gather are collected and made available here: https://drive.google.com/drive/folders/13ujtnaojbntYlK_TM_B_tLFVOQn6H6in?usp=sharing

              jskarvad Jaroslav Skarvada
              midu@redhat.com Mihai IDU
              Jaroslav Skarvada Jaroslav Skarvada
              Robin Hack Robin Hack
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

                Created:
                Updated:
                Resolved: