Node tuning profiles are not removed from hypershift guest clusters after worker nodes are removed. The result is that the profile persists despite not having a matching node present within the cluster.

This is easy to reproduce in a hypershift guest cluster using the kubevirt platform. It is unclear at this time if the issue is platform specific, or occurs for all hypershift platforms. 

4.14

100%

1. create a hypershift/kubevirt guest cluster with 2 replicas
2. delete one of the machine.cluster.x-k8s.io objects
3. a new node will replace the deleted one, but the NTO profile will persist for the old node. 

Node vossel1-m26td is removed, but the profile persists.

[dvossel@fedora hypershift]$ oc --kubeconfig test-kubeconfig get profiles -A
NAMESPACE                                NAME            TUNED            APPLIED   DEGRADED   AGE
openshift-cluster-node-tuning-operator   vossel1-m26td   openshift-node   True      False      3h5m
openshift-cluster-node-tuning-operator   vossel1-mc2wn   openshift-node   True      False      3h19m
openshift-cluster-node-tuning-operator   vossel1-sw44t   openshift-node   True      False      15m
[dvossel@fedora hypershift]$ oc --kubeconfig test-kubeconfig get nodes
NAME            STATUS   ROLES    AGE     VERSION
vossel1-mc2wn   Ready    worker   3h19m   v1.26.2+d2e245f
vossel1-sw44t   Ready    worker   15m     v1.26.2+d2e245f

No NTO profilei should exist for the non-existent node vossel1-m26td

nto logs

[dvossel@fedora hypershift]$ oc logs -n clusters-vossel1 cluster-node-tuning-operator-7c449dbd74-7fnq9
I0501 15:46:11.397637       1 main.go:71] Go Version: go1.19.6
I0501 15:46:11.397714       1 main.go:72] Go OS/Arch: linux/amd64
I0501 15:46:11.397718       1 main.go:73] node-tuning Version: v4.14.0-202304290616.p0.g947fd2d.assembly.stream-0-gbbcefec-dirty
W0501 15:46:11.408893       1 leaderelection.go:34] unable to get cluster infrastructure status, using HA cluster values for leader election: infrastructures.config.openshift.io "cluster" not found
I0501 15:46:11.419436       1 leaderelection.go:248] attempting to acquire leader lease openshift-cluster-node-tuning-operator/node-tuning-operator-lock...
E0501 15:46:11.474040       1 leaderelection.go:334] error initially creating leader election record: namespaces "openshift-cluster-node-tuning-operator" not found
I0501 15:47:08.550950       1 leaderelection.go:258] successfully acquired lease openshift-cluster-node-tuning-operator/node-tuning-operator-lock
I0501 15:47:08.551123       1 controller.go:1287] starting Tuned controller
I0501 15:47:08.551943       1 server.go:102] starting metrics server
I0501 15:47:08.652036       1 controller.go:1387] started events processor/controller
I0501 15:47:08.673663       1 controller.go:535] created Tuned rendered
I0501 15:51:56.615142       1 controller.go:681] created profile vossel1-jkqc9 [openshift-node]
I0501 15:51:56.638474       1 controller.go:764] updated profile vossel1-jkqc9 [openshift-node]
I0501 15:52:07.993941       1 controller.go:681] created profile vossel1-mc2wn [openshift-node]
I0501 15:52:08.008909       1 controller.go:764] updated profile vossel1-mc2wn [openshift-node]
I0501 16:05:55.243794       1 controller.go:681] created profile vossel1-m26td [openshift-node]
I0501 16:05:55.257475       1 controller.go:764] updated profile vossel1-m26td [openshift-node]
I0501 18:55:46.771467       1 controller.go:681] created profile vossel1-sw44t [openshift-node]
I0501 18:55:46.788263       1 controller.go:764] updated profile vossel1-sw44t [openshift-node]