Version annotation on Windows BYOH node is failing since HNS cleanup, the node IP cannot be found 

    10.16.0-a2d712a, 10.17.0-b7d6595

    100%

    1. create a Windows BYOH node 
    2. Change Windows node, such as remove annotation: windowsmachineconfig.openshift.io/version
# oc edit node ip-10-0-159-151.us-east-2.compute.internal
     3. Wait for node to be reconciled 
    

    Node does not appear after version annotation
{"level":"info","ts":"2024-08-01T11:32:59Z","logger":"wc 10.0.9.247","msg":"removing HNS networks"}
{"level":"error","ts":"2024-08-01T11:32:59Z","logger":"wc 10.0.9.247","msg":"error removing %s HNS network","network":"BaseOVNKubernetesHybridOverlayNetwork","error":"failed to remove BaseOVNKubernetesHybridOverlayNetwork HNS network with output: : error running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"Get-HnsNetwork | where { $_.Name -eq 'BaseOVNKubernetesHybridOverlayNetwork'} | Remove-HnsNetwork;\": read tcp 10.0.42.255:49660->10.0.9.247:22: read: connection reset by peer","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHNSNetworksAreRemoved.func1\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:994\nk8s.io/apimachinery/pkg/util/wait.PollImmediate.ConditionFunc.WithContext.func1\n\t/remote-source/build/windows-machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:109\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/remote-source/build/windows-machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:154\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/remote-source/build/windows-machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:245\nk8s.io/apimachinery/pkg/util/wait.PollImmediateWithContext\n\t/remote-source/build/windows-machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:175\nk8s.io/apimachinery/pkg/util/wait.PollImmediate\n\t/remote-source/build/windows-machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/wait/poll.go:157\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureHNSNetworksAreRemoved\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:991\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).RemoveFilesAndNetworks\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:519\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Deconfigure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:542\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:93\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).ensureInstancesAreUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:316\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).reconcileNodes\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:281\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:191\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
{"level":"info","ts":"2024-08-01T11:36:01Z","logger":"wc 10.0.9.247","msg":"removing directories"}
{"level":"info","ts":"2024-08-01T11:36:01Z","logger":"wc 10.0.9.247","msg":"removing directories"}
{"level":"info","ts":"2024-08-01T11:37:10Z","logger":"nc 10.0.9.247","msg":"instance has been deconfigured","node":"ip-10-0-9-247.us-east-2.compute.internal"}
{"level":"info","ts":"2024-08-01T11:37:10Z","logger":"nc 10.0.9.247","msg":"instance has been deconfigured","node":"ip-10-0-9-247.us-east-2.compute.internal"}
{"level":"info","ts":"2024-08-01T11:37:31Z","logger":"wc 10.0.9.247","msg":"configuring"}
{"level":"info","ts":"2024-08-01T11:37:31Z","logger":"wc 10.0.9.247","msg":"configuring"}
{"level":"info","ts":"2024-08-01T11:37:55Z","logger":"wc 10.0.9.247","msg":"transferring files"}
{"level":"info","ts":"2024-08-01T11:37:55Z","logger":"wc 10.0.9.247","msg":"transferring files"}
{"level":"info","ts":"2024-08-01T11:39:55Z","logger":"wc 10.0.9.247","msg":"configured","service":"windows-instance-config-daemon","args":"controller --windows-service --log-dir C:\\var\\log\\wicd --kubeconfig C:\\k\\wicd-kubeconfig --namespace openshift-windows-machine-config-operator --ca-bundle C:\\k\\ca-bundle.crt"}
{"level":"info","ts":"2024-08-01T11:39:55Z","logger":"wc 10.0.9.247","msg":"configured","service":"windows-instance-config-daemon","args":"controller --windows-service --log-dir C:\\var\\log\\wicd --kubeconfig C:\\k\\wicd-kubeconfig --namespace openshift-windows-machine-config-operator --ca-bundle C:\\k\\ca-bundle.crt"}
{"level":"info","ts":"2024-08-01T11:39:58Z","logger":"wc 10.0.9.247","msg":"rebooting instance"}
{"level":"info","ts":"2024-08-01T11:39:58Z","logger":"wc 10.0.9.247","msg":"rebooting instance"}
{"level":"info","ts":"2024-08-01T11:41:45Z","logger":"controllers.certificatesigningrequests","msg":"CSR contents are invalid for approval by WMCO","CSR":"csr-pv4mh"}
{"level":"info","ts":"2024-08-01T11:41:45Z","logger":"controllers.certificatesigningrequests","msg":"CSR contents are invalid for approval by WMCO","CSR":"csr-pv4mh"}
{"level":"info","ts":"2024-08-01T11:42:10Z","logger":"nc 10.0.9.247","msg":"Unable to mark node as NotReady","error":"error ensuring windows-instance-config-daemon Windows service is removed: error checking if windows-instance-config-daemon Windows service exists: error running sc.exe qc windows-instance-config-daemon: read tcp 10.0.42.255:33046->10.0.9.247:22: read: connection reset by peer"}
{"level":"error","ts":"2024-08-01T11:42:10Z","msg":"Reconciler error","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","ConfigMap":{"name":"windows-instances","namespace":"openshift-windows-machine-config-operator"},"namespace":"openshift-windows-machine-config-operator","name":"windows-instances","reconcileID":"029b449b-91cc-42eb-b2b5-3c10d608b4a3","error":"error configuring host with address 10.0.9.247: error waiting for proper windowsmachineconfig.openshift.io/version annotation for node ip-10-0-9-247.us-east-2.compute.internal: timeout waiting for windowsmachineconfig.openshift.io/version {"level":"info","ts":"2024-08-01T11:42:10Z","logger":"nc 10.0.9.247","msg":"Unable to mark node as NotReady","error":"error ensuring windows-instance-config-daemon Windows service is removed: error checking if windows-instance-config-daemon Windows service exists: error running sc.exe qc windows-instance-config-daemon: read tcp 10.0.42.255:33046->10.0.9.247:22: read: connection reset by peer"}
{"level":"error","ts":"2024-08-01T11:42:10Z","msg":"Reconciler error","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","ConfigMap":{"name":"windows-instances","namespace":"openshift-windows-machine-config-operator"},"namespace":"openshift-windows-machine-config-operator","name":"windows-instances","reconcileID":"029b449b-91cc-42eb-b2b5-3c10d608b4a3","error":"error configuring host with address 10.0.9.247: error waiting for proper windowsmachineconfig.openshift.io/version annotation for node ip-10-0-9-247.us-east-2.compute.internal: timeout waiting for windowsmachineconfig.openshift.io/version and windowsmachineconfig.openshift.io/desired-version annotations to match on node ip-10-0-9-247.us-east-2.compute.internal: Node \"ip-10-0-9-247.us-east-2.compute.internal\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
and windowsmachineconfig.openshift.io/desired-version annotations to match on node ip-10-0-9-247.us-east-2.compute.internal: Node \"ip-10-0-9-247.us-east-2.compute.internal\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222

    BYOH node should appear after version annotation