Creating a new MCP  with a machineConfigSelector that includes worker causes NTO-managed MachineConfigs (with tuned.openshift.io/generated-by-controller-version annotation) to be deleted, triggering the worker pool to re-render with a new hash and causing unnecessary worker node reboots. 

    4.21.0-rc.1

Always

1. Create the machine config without the annotation

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 50-nto-worker
spec:
  kernelArguments:
  - skew_tick=1

2. Wait for the nodes to reboot

3. Edit the 50-nto-worker machine config to include the annotation

 annotations:
   tuned.openshift.io/generated-by-controller-version: 78f0071ec

4. Apply the mcp-test.yaml

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: mcp-test
  labels:
    machineconfiguration.openshift.io/role: mcp-test
spec:
  machineConfigSelector:
    matchExpressions:
      - {
          key: machineconfiguration.openshift.io/role,
          operator: In,
          values: [mcp-test, worker]
        }
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/mcp-test: ""

5. The machine config 50-nto-worker will get deleted by applying the mcp causing the nodes to reboot unintentionally.

The machine config 50-nto-worker is being deleted causing the worker nodes to reboot

The machine config 50-nto-worker should not be deleted when this mcp is created

This bug was found in this test case in the NROP serial suite https://github.com/openshift-kni/numaresources-operator/blob/main/test/e2e/serial/tests/configuration.go#L1015 as we are using a performanceprofile which applies the 50-nto-worker machine config with this annotation

More info about the reproduction of this:

This logs from the cluster is after editing the machine config to include the annotation and then applying the mcp-test we can see it does the reboot of the nodes

root@sealusa58:~# oc get no,mcp,mc

NAME                                                            STATUS                     ROLES                  AGE   VERSIONnode/ocp4211130662-ctlplane-0.libvirt.lab.eng.tlv2.redhat.com   Ready                      control-plane,master   20h   v1.34.2node/ocp4211130662-ctlplane-1.libvirt.lab.eng.tlv2.redhat.com   Ready                      control-plane,master   20h   v1.34.2node/ocp4211130662-ctlplane-2.libvirt.lab.eng.tlv2.redhat.com   Ready                      control-plane,master   20h   v1.34.2node/ocp4211130662-worker-0.libvirt.lab.eng.tlv2.redhat.com     Ready                      worker                 20h   v1.34.2node/ocp4211130662-worker-1.libvirt.lab.eng.tlv2.redhat.com     Ready,SchedulingDisabled   worker                 20h   v1.34.2

NAME                                                           CONFIG                                               UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGEmachineconfigpool.machineconfiguration.openshift.io/master     rendered-master-41d3e81c37aefccddba1b43e77954dfb     True      False      False      3              3                   3                     0                      20hmachineconfigpool.machineconfiguration.openshift.io/mcp-test   rendered-mcp-test-6ba9185a7ead1eb0fbd9bd25ff770d4a   True      False      False      0              0                   0                     0                      50smachineconfigpool.machineconfiguration.openshift.io/worker     rendered-worker-8d432660234025fc76ed73c8458af831     False     True       False      2              0                   0                     0                      20h

NAME                                                                                                 GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGEmachineconfig.machineconfiguration.openshift.io/00-master                                            251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/00-worker                                            251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/01-master-container-runtime                          251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/01-master-kubelet                                    251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/01-worker-container-runtime                          251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/01-worker-kubelet                                    251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/97-master-generated-kubelet                          251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/97-worker-generated-kubelet                          251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/98-master-generated-kubelet                          251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/98-worker-generated-kubelet                          251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/99-iptables                                                                                     3.2.0             20hmachineconfig.machineconfiguration.openshift.io/99-master-generated-registries                       251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/99-master-registries                                                                            2.2.0             19hmachineconfig.machineconfiguration.openshift.io/99-master-ssh                                                                                   3.2.0             20hmachineconfig.machineconfiguration.openshift.io/99-worker-generated-registries                       251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/99-worker-registries                                                                            2.2.0             19hmachineconfig.machineconfiguration.openshift.io/99-worker-ssh                                                                                   3.2.0             20hmachineconfig.machineconfiguration.openshift.io/rendered-master-41d3e81c37aefccddba1b43e77954dfb     251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             19hmachineconfig.machineconfiguration.openshift.io/rendered-master-fde7326e11c0d9a3ae9308b4fdbc82f3     251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/rendered-mcp-test-6ba9185a7ead1eb0fbd9bd25ff770d4a   251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             45smachineconfig.machineconfiguration.openshift.io/rendered-worker-142b7ee9d18000fc234fd774a5057089     251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             20hmachineconfig.machineconfiguration.openshift.io/rendered-worker-6ba9185a7ead1eb0fbd9bd25ff770d4a     251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             19hmachineconfig.machineconfiguration.openshift.io/rendered-worker-8d432660234025fc76ed73c8458af831     251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             62mmachineconfig.machineconfiguration.openshift.io/rendered-worker-a3d0116f480d8f175ad25c20e543e1dc     251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             19hmachineconfig.machineconfiguration.openshift.io/rendered-worker-ab91df1a5fe19714ff65ca69d1f26128     251c4bdf92564d0a97ced5459f7a809e8d3ef11c   3.5.0             158m