Having a 4.14 OCP cluster with BYOH and Machine Windows Containers nodes, when modifying the the version annotation windowsmachineconfig.openshift.io/version. If the version set does not exist, for example:
oc annotate node ip-10-0-148-206.us-east-2.compute.internal --overwrite windowsmachineconfig.openshift.io/version=invalidVersion

The impacted BYOH/Machine node will hang on Ready,Scheduling disabled and won't be able to leave that state as WMCO is trying to allocate the configmap windows-services-invalidVersion, which does not exist:

[jfrancoa@localhost openshift-tests-private]$ oc get cm windows-instances -n openshift-windows-machine-config-operator -o yaml
apiVersion: v1
data:
  10.0.148.206: username=Administrator
kind: ConfigMap
metadata:
  creationTimestamp: "2023-05-18T07:31:13Z"
  name: windows-instances
  namespace: openshift-windows-machine-config-operator
  resourceVersion: "60960"
  uid: ab745d75-6f53-4919-b930-4edc88e5016d
[jfrancoa@localhost openshift-tests-private]$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-136-0.us-east-2.compute.internal Ready control-plane,master 135m v1.27.1+20a4409 10.0.136.0 <none> Red Hat Enterprise Linux CoreOS 414.92.202305162029-0 (Plow) 5.14.0-284.13.1.el9_2.x86_64 cri-o://1.27.0-6.rhaos4.14.git81ac4ce.el9
ip-10-0-137-97.us-east-2.compute.internal Ready worker 88m v1.26.2+0f23833 10.0.137.97 <none> Windows Server 2019 Datacenter 10.0.17763.4252 containerd://1.7.0
ip-10-0-139-10.us-east-2.compute.internal Ready worker 126m v1.27.1+20a4409 10.0.139.10 <none> Red Hat Enterprise Linux CoreOS 414.92.202305162029-0 (Plow) 5.14.0-284.13.1.el9_2.x86_64 cri-o://1.27.0-6.rhaos4.14.git81ac4ce.el9
ip-10-0-148-206.us-east-2.compute.internal Ready,SchedulingDisabled worker 23m v1.26.2+0f23833 10.0.148.206 <none> Windows Server 2019 Datacenter 10.0.17763.4252 containerd://1.7.0
ip-10-0-149-147.us-east-2.compute.internal Ready worker 83m v1.26.2+0f23833 10.0.149.147 <none> Windows Server 2019 Datacenter 10.0.17763.4252 containerd://1.7.0
ip-10-0-185-94.us-east-2.compute.internal Ready worker 127m v1.27.1+20a4409 10.0.185.94 <none> Red Hat Enterprise Linux CoreOS 414.92.202305162029-0 (Plow) 5.14.0-284.13.1.el9_2.x86_64 cri-o://1.27.0-6.rhaos4.14.git81ac4ce.el9
ip-10-0-191-102.us-east-2.compute.internal Ready control-plane,master 134m v1.27.1+20a4409 10.0.191.102 <none> Red Hat Enterprise Linux CoreOS 414.92.202305162029-0 (Plow) 5.14.0-284.13.1.el9_2.x86_64 cri-o://1.27.0-6.rhaos4.14.git81ac4ce.el9
ip-10-0-212-18.us-east-2.compute.internal Ready worker 124m v1.27.1+20a4409 10.0.212.18 <none> Red Hat Enterprise Linux CoreOS 414.92.202305162029-0 (Plow) 5.14.0-284.13.1.el9_2.x86_64 cri-o://1.27.0-6.rhaos4.14.git81ac4ce.el9
ip-10-0-213-56.us-east-2.compute.internal Ready control-plane,master 134m v1.27.1+20a4409 10.0.213.56 <none> Red Hat Enterprise Linux CoreOS 414.92.202305162029-0 (Plow) 5.14.0-284.13.1.el9_2.x86_64 cri-o://1.27.0-6.rhaos4.14.git81ac4ce.el9

WMCO logs:

{"level":"info","ts":"2023-05-18T07:44:13Z","logger":"nc 10.0.148.206","msg":"instance has been configured as a worker node","version":"9.0.0-0ecb2e1"}
{"level":"info","ts":"2023-05-18T07:44:13Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
{"level":"info","ts":"2023-05-18T07:45:13Z","logger":"controllers.configmap","msg":"processing","instances in":"windows-instances"}
{"level":"info","ts":"2023-05-18T07:45:13Z","logger":"controllers.configmap","msg":"instance is up to date","node":"ip-10-0-148-206.us-east-2.compute.internal","version":"9.0.0-0ecb2e1"}
{"level":"info","ts":"2023-05-18T07:45:13Z","logger":"metrics","msg":"Prometheus configured","endpoints":"windows-exporter","port":9182,"name":"metrics"}
{"level":"info","ts":"2023-05-18T07:46:05Z","logger":"controllers.configmap","msg":"processing","instances in":"windows-instances"}
{"level":"info","ts":"2023-05-18T07:46:06Z","logger":"controllers.configmap","msg":"instance requires upgrade","node":"ip-10-0-148-206.us-east-2.compute.internal","version":"invalidVersion","expected version":"9.0.0-0ecb2e1"}
{"level":"info","ts":"2023-05-18T07:46:16Z","logger":"nc 10.0.148.206","msg":"evicting pod winc-42484/win-webserver-768b7bc78d-smjhc\n"}
{"level":"info","ts":"2023-05-18T07:46:16Z","logger":"nc 10.0.148.206","msg":"evicting pod winc-42484/win-webserver-768b7bc78d-2vwxs\n"}
{"level":"info","ts":"2023-05-18T07:46:16Z","logger":"nc 10.0.148.206","msg":"evicting pod winc-42484/win-webserver-768b7bc78d-857wc\n"}
{"level":"info","ts":"2023-05-18T07:46:16Z","logger":"nc 10.0.148.206","msg":"evicting pod winc-42484/win-webserver-768b7bc78d-gq7g5\n"}
{"level":"info","ts":"2023-05-18T07:46:16Z","logger":"nc 10.0.148.206","msg":"evicting pod winc-42484/win-webserver-768b7bc78d-pzkdd\n"}
{"level":"info","ts":"2023-05-18T07:46:16Z","logger":"wc 10.0.148.206","msg":"deconfiguring"}
{"level":"info","ts":"2023-05-18T07:46:47Z","logger":"wc 10.0.148.206","msg":"deconfigured","service":"windows-instance-config-daemon"}
{"level":"error","ts":"2023-05-18T07:46:52Z","logger":"wc 10.0.148.206","msg":"error running","cmd":"powershell.exe -NonInteractive -ExecutionPolicy Bypass \"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace openshi
ft-windows-machine-config-operator\"","out":"F0518 07:46:52.825267 2756 cleanup.go:51] configmaps \"windows-services-invalidVersion\" not found\n","error":"Process exited with status 1","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/win
dows.(*windows).Run\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:381\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).RunWICDCleanup\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/
windows.go:408\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Deconfigure\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:418\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeC
onfig).Deconfigure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:485\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-co
nfig-operator/controllers/controllers.go:79\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).ensureInstancesAreUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:314\ngithub
.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).reconcileNodes\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:279\ngithub.com/openshift/windows-machine-config-operator/controllers.(*Co
nfigMapReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/configmap_controller.go:189\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/ve
ndor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/
internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}
{"level":"info","ts":"2023-05-18T07:46:52Z","logger":"wc 10.0.148.206","msg":"failed to cleanup node","command":"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace openshift-windows-machine-config-operator","output":
"F0518 07:46:52.825267 2756 cleanup.go:51] configmaps \"windows-services-invalidVersion\" not found\n"}
{"level":"error","ts":"2023-05-18T07:46:52Z","msg":"Reconciler error","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","ConfigMap":{"name":"windows-instances","namespace":"openshift-windows-machine-config-operator"},"namespace":"openshif
t-windows-machine-config-operator","name":"windows-instances","reconcileID":"6cf1733d-f108-43b9-832e-ee4400e00eef","error":"error configuring host with address 10.0.148.206: error deconfiguring instance: unable to cleanup the Windows instance: error running power
shell.exe -NonInteractive -ExecutionPolicy Bypass \"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace openshift-windows-machine-config-operator\": Process exited with status 1","stacktrace":"sigs.k8s.io/controller-r
untime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Control
ler).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/
build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}
{"level":"info","ts":"2023-05-18T07:46:52Z","logger":"controllers.configmap","msg":"processing","instances in":"windows-instances"}
{"level":"info","ts":"2023-05-18T07:46:53Z","logger":"controllers.configmap","msg":"instance requires upgrade","node":"ip-10-0-148-206.us-east-2.compute.internal","version":"invalidVersion","expected version":"9.0.0-0ecb2e1"}

[jfrancoa@localhost openshift-tests-private]$ oc get cm -n openshift-windows-machine-config-operator
NAME DATA AGE
kube-root-ca.crt 1 102m
openshift-service-ca.crt 1 102m
windows-instances 1 35m
windows-machine-config-operator-lock 0 101m
windows-services-9.0.0-0ecb2e1 2 101m
[jfrancoa@localhost openshift-tests-private]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.14.0-0.nightly-2023-05-18-040905 True False 113m Cluster version is 4.14.0-0.nightly-2023-05-18-040905

Always

1. Deploy an OCP 4.14 cluster. Install WMCO on it.
2. Create a BYOH or Machine Windows node
3. Modify the version annotation and set it to invalidVersion: oc annotate node ip-10-0-148-206.us-east-2.compute.internal --overwrite windowsmachineconfig.openshift.io/version=invalidVersion
4. Wait for the node to reconcile

The node does not reconcile and hangs in Ready,SchedulingDisabled for ever

The node reconciles and restores back the invalid version annotation.

This is a regression. This functionality was working in all previous versions.