Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-63244

PerformanceProfile config is generating extra rendered-config after cluster is upgraded

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.16
    • Node Tuning Operator
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      When applied the Machine config, Container runtime config, Performance Profile config on cluster and later upgrade the cluster, after upgrade able to see a rendered-xx config generated and after few minutes it is re-generated again. 

      Version-Release number of selected component (if applicable):

          

      How reproducible:

         Always

      Steps to Reproduce:

       Apply the below configs

      oc create -f - << EOF
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: worker
        name: test-file
      spec:
        config:
          ignition:
            version: 3.4.0
          storage:
            files:
            - contents:
                source: data:text/plain;charset=utf-8;base64,dGVzdA==
              filesystem: root
              mode: 420
              path: /etc/test-file.test
      EOF
      machineconfig.machineconfiguration.openshift.io/test-file created

      Check rendered config is generated

      oc get mc --sort-by=.metadata.creationTimestamp
      test-file                                                                                                3.4.0             30s
      rendered-worker-54dd08819c8a40b7a1cdeb98edc5019a              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             25s 

      again apply the below config

      oc apply -f - << EOF
      apiVersion: machineconfiguration.openshift.io/v1
      kind: ContainerRuntimeConfig
      metadata:
        name: worker-crun-runtime
      spec:
        machineConfigPoolSelector:
          matchLabels:
            pools.operator.machineconfiguration.openshift.io/worker: ""
        containerRuntimeConfig:
          defaultRuntime: crun
          logLevel: debug
      EOF
      containerruntimeconfig.machineconfiguration.openshift.io/worker-crun-runtime created

      Check rendered generated

      oc get mc --sort-by=.metadata.creationTimestamp
      NAME                                                          GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE 
      ....
      test-file                                                                                                3.4.0             30s
      rendered-worker-54dd08819c8a40b7a1cdeb98edc5019a              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             25s
      99-worker-generated-containerruntime                          f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             9s
      rendered-worker-70ba6eb64491ee4c6b9dea55faeca3e5              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             4s

      Apply the below Performance Profile Config

      oc apply -f - << EOF
      apiVersion: performance.openshift.io/v2
      kind: PerformanceProfile
      metadata:
        labels:
          machineconfiguration.openshift.io/role: worker
        name: ovs-dpdk-worker
      annotations:
        kubeletconfig.experimental: |
          cpuManagerPolicyOptions: {"full-pcpus-only": true, "strict-cpu-reservation": "true"}
      spec:
        cpu:
          isolated: "2-3"
          reserved: "0-1"
        nodeSelector:
          node-role.kubernetes.io/worker: ''
        numa:
          topologyPolicy: "restricted"
        realTimeKernel:
          enabled: true
      EOF
      performanceprofile.performance.openshift.io/ovs-dpdk-worker created
      oc get mc --sort-by=.metadata.creationTimestamp 
      NAME                                                          GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
      ....
      test-file                                                                                                3.4.0             62s
      rendered-worker-54dd08819c8a40b7a1cdeb98edc5019a              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             57s
      99-worker-generated-containerruntime                          f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             41s
      rendered-worker-70ba6eb64491ee4c6b9dea55faeca3e5              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             36s
      50-performance-ovs-dpdk-worker                                                                           3.2.0             26s
      99-worker-generated-kubelet                                   f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             26s
      50-nto-worker                                                                                                              25s
      rendered-worker-89ec5fb43c430a4320a54dfeff938451              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             21s  
      
      

      Wait for MCP update to complete

      $ oc get mcp
      NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master   rendered-master-0eb42ead0f22d08cf58abdbece40327b   True      False      False      3              3                   3                     0                      142m
      worker   rendered-worker-59e5af6b8629e7228da45cb432dd689a   False     True       False      3              2                   2                     0                      142m

      Then upgrade the cluster

      $ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release:4.18.0-0.nightly-2025-10-14-211520 --allow-explicit-upgrade  --force 
       $ oc get clusterversion  NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS version   4.17.0-0.nightly-2025-10-14-161700   True        True          5m56s   Working towards 4.18.0-0.nightly-2025-10-14-211520: 111 of 904 done (12% complete), waiting on etcd, kube-apiserver
      

      After upgrade able to see this

      oc get mc --sort-by=.metadata.creationTimestamp
      NAME                                                          GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
      ....
      test-file                                                                                                3.4.0             4h22m
      rendered-worker-54dd08819c8a40b7a1cdeb98edc5019a              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             4h22m
      99-worker-generated-containerruntime                          0f7ca82c30f45e9c7d26279e920b63bbf233e130   3.4.0             4h21m
      rendered-worker-70ba6eb64491ee4c6b9dea55faeca3e5              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             4h21m
      99-worker-generated-kubelet                                   0f7ca82c30f45e9c7d26279e920b63bbf233e130   3.4.0             4h21m
      50-performance-ovs-dpdk-worker                                                                           3.2.0             4h21m
      50-nto-worker                                                                                                              4h21m
      rendered-worker-89ec5fb43c430a4320a54dfeff938451              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             4h21m
      rendered-worker-6af1abdc8ea035d41fa3ec0627ca8bd7              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             3h26m
      97-master-generated-kubelet                                   0f7ca82c30f45e9c7d26279e920b63bbf233e130   3.4.0             3h11m
      97-worker-generated-kubelet                                   0f7ca82c30f45e9c7d26279e920b63bbf233e130   3.4.0             3h11m
      rendered-worker-50b0c7e52c25b6a78505b4609e8657fc              0f7ca82c30f45e9c7d26279e920b63bbf233e130   3.4.0             3h11m
      rendered-master-c882073a77f344afe0d4561b561db29a              0f7ca82c30f45e9c7d26279e920b63bbf233e130   3.4.0             3h11m 

      At [3h26m] rendered-worker-xx is generated and again later at  [3h11m] the rendered-worker-xx is generated

      Actual results:

       

      Expected results:

           Before 99-xx-generated-kubelet no endered-worker-xx should be generated

      Additional info:

      I tested on cluster without Performance Config no extra rendered-xx was generated

      $ oc get mc --sort-by=.metadata.creationTimestamp
      NAME                                                          GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
      ...
      test-file                                                                                                3.4.0             4h
      rendered-worker-1fe32d3722dab050f5fceea1db9d82d4              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             4h
      99-worker-generated-containerruntime                          86f5e905d071da6377abca62c135c792f706d1d0   3.4.0             3h59m
      rendered-worker-3a492f0513745bf903456d4e1f0743fb              f367312c9951268ea8355e5ebb854af05cb4256f   3.4.0             3h59m
      97-master-generated-kubelet                                   86f5e905d071da6377abca62c135c792f706d1d0   3.4.0             95m
      97-worker-generated-kubelet                                   86f5e905d071da6377abca62c135c792f706d1d0   3.4.0             95m
      rendered-worker-732bf7afc89bba3a5b6fe2edfd146547              86f5e905d071da6377abca62c135c792f706d1d0   3.4.0             95m
      rendered-master-4c3ba0b19ca23fd892934214e033f7d7              86f5e905d071da6377abca62c135c792f706d1d0   3.4.0             95m

              team-nto Team NTO
              rh-ee-ptalgulk Prachiti Talgulkar
              None
              None
              Liquan Cui Liquan Cui
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: