Installing WMCO on a different namespace than windows-machine-config-operator is failing, nodes are missing machines Stuck in provisioning state. 

9.0.0-9456533

100% on all platforms

1. Deploy IPI on AWS with WMCO 9.0.0 (windows-services-9.0.0-9456533) on the default namespace openshift-windows-machine-config-operator
2. Create a Windows MachineSet (Windows 2019), scale it up to two and wait for the nodes to be in Ready state
3. Create workloads that will land in those two Windows Machine nodes
4. Scale down the Windows MachineSet to 0
5. Uninstall WMCO by: deleting the subscription, the operator group and the namespace
6. Install back WMCO in winc-namespace-test: create a winc-namespace-test, create the WMCO secret on the namespace, create operator group and subscription
7. Wait for WMCO to be up and running
8. Scale up the Windows MachineSet back to 2
9. Wait for the nodes to be in ready state. But the nodes never come up and the log Ronnie mentioned originally appears.

Node are not scaling up with the new namespace, machines stuck in provisioning

WMCO shuold work on different namespace than windows-machine-config-operator

{"level":"info","ts":"2023-11-09T15:10:17Z","logger":"wc 10.0.5.101","msg":"configuring"}
{"level":"error","ts":"2023-11-09T15:10:29Z","logger":"wc 10.0.5.101","msg":"error running","cmd":"powershell.exe -NonInteractive -ExecutionPolicy Bypass \"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace winc-namespace-test\"","out":"I1109 15:10:29.196654     636 cleanup.go:55] no associated node found\nF1109 15:10:29.262184     636 cleanup.go:51] cannot get latest services ConfigMap from namespace winc-namespace-test: configmaps is forbidden: User \"system:serviceaccount:winc-namespace-test:windows-instance-config-daemon\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"winc-namespace-test\"\n","error":"Process exited with status 1","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Run\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:399\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).RunWICDCleanup\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:437\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:458\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:165\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:110\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:426\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226"}
{"level":"info","ts":"2023-11-09T15:10:29Z","logger":"wc 10.0.5.101","msg":"failed to cleanup node","command":"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace winc-namespace-test","output":"I1109 15:10:29.196654     636 cleanup.go:55] no associated node found\nF1109 15:10:29.262184     636 cleanup.go:51] cannot get latest services ConfigMap from namespace winc-namespace-test: configmaps is forbidden: User \"system:serviceaccount:winc-namespace-test:windows-instance-config-daemon\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"winc-namespace-test\"\n"}

oc get cm -n openshift-windows-machine-config-operator windows-services-9.0.0-9456533 -oyaml
apiVersion: v1
data:
  files: '[]'
  services: '[{"name":"containerd","path":"C:\\k\\containerd\\containerd.exe --config
    C:\\k\\containerd\\containerd_conf.toml --log-file C:\\var\\log\\containerd\\containerd.log
    --run-service --log-level info","powershellPreScripts":[{"path":"C:\\Temp\\windows-defender-exclusion.ps1
    -BinPath C:\\k\\containerd\\containerd.exe"}],"bootstrap":true,"priority":0},{"name":"kubelet","path":"C:\\k\\kube-log-runner.exe
    -log-file=C:\\var\\log\\kubelet\\kubelet.log C:\\k\\kubelet.exe --config=C:\\k\\kubelet.conf
    --bootstrap-kubeconfig=C:\\k\\bootstrap-kubeconfig --kubeconfig=C:\\k\\kubeconfig
    --cert-dir=c:\\var\\lib\\kubelet\\pki\\ --windows-service --register-with-taints=os=Windows:NoSchedule
    --node-labels=node.openshift.io/os_id=Windows --resolv-conf= --windows-priorityclass=ABOVE_NORMAL_PRIORITY_CLASS
    --v=2 --cloud-provider=external --hostname-override=HOSTNAME_OVERRIDE","powershellPreScripts":[{"variableName":"HOSTNAME_OVERRIDE","path":"Invoke-RestMethod
    -UseBasicParsing -Uri http://169.254.169.254/latest/meta-data/local-hostname"}],"dependencies":["containerd"],"bootstrap":true,"priority":1},{"name":"windows_exporter","path":"C:\\k\\windows_exporter.exe
    --collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info","bootstrap":false,"priority":2},{"name":"hybrid-overlay-node","path":"C:\\k\\hybrid-overlay-node.exe
    --node NODE_NAME --bootstrap-kubeconfig=C:\\k\\kubeconfig --cert-dir=C:\\k\\cni\\config
    --cert-duration=24h --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log","nodeVariablesInCommand":[{"name":"NODE_NAME","nodeObjectJsonPath":"{.metadata.name}"}],"dependencies":["kubelet"],"bootstrap":false,"priority":2},{"name":"csi-proxy","path":"C:\\k\\csi-proxy.exe
    -log_file=C:\\var\\log\\csi-proxy\\csi-proxy.log -logtostderr=false -windows-service
    --v=2","bootstrap":false,"priority":2},{"name":"kube-proxy","path":"C:\\k\\kube-log-runner.exe
    -log-file=C:\\var\\log\\kube-proxy\\kube-proxy.log C:\\k\\kube-proxy.exe --windows-service
    --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=NODE_NAME
    --kubeconfig=C:\\k\\kubeconfig --cluster-cidr=NODE_SUBNET --network-name=OVNKubernetesHybridOverlayNetwork
    --source-vip=ENDPOINT_IP --enable-dsr=false --v=2","nodeVariablesInCommand":[{"name":"NODE_NAME","nodeObjectJsonPath":"{.metadata.name}"},{"name":"NODE_SUBNET","nodeObjectJsonPath":"{.metadata.annotations.k8s\\.ovn\\.org/hybrid-overlay-node-subnet}"}],"powershellPreScripts":[{"variableName":"ENDPOINT_IP","path":"C:\\Temp\\network-conf.ps1"}],"dependencies":["hybrid-overlay-node"],"bootstrap":false,"priority":3}]'
  watchedEnvironmentVars: '["HTTP_PROXY","HTTPS_PROXY","NO_PROXY"]'
immutable: true
kind: ConfigMap
metadata:
  creationTimestamp: "2023-11-09T15:35:29Z"
  name: windows-services-9.0.0-9456533
  namespace: openshift-windows-machine-config-operator
  resourceVersion: "115700"
  uid: 7f4c4f68-5aa8-40c3-a746-09636d1235c6

oc get cm -n winc-namespace-test windows-services-9.0.0-9456533 -oyaml
apiVersion: v1
data:
  files: '[]'
  services: '[{"name":"containerd","path":"C:\\k\\containerd\\containerd.exe --config
    C:\\k\\containerd\\containerd_conf.toml --log-file C:\\var\\log\\containerd\\containerd.log
    --run-service --log-level info","powershellPreScripts":[{"path":"C:\\Temp\\windows-defender-exclusion.ps1
    -BinPath C:\\k\\containerd\\containerd.exe"}],"bootstrap":true,"priority":0},{"name":"kubelet","path":"C:\\k\\kube-log-runner.exe
    -log-file=C:\\var\\log\\kubelet\\kubelet.log C:\\k\\kubelet.exe --config=C:\\k\\kubelet.conf
    --bootstrap-kubeconfig=C:\\k\\bootstrap-kubeconfig --kubeconfig=C:\\k\\kubeconfig
    --cert-dir=c:\\var\\lib\\kubelet\\pki\\ --windows-service --register-with-taints=os=Windows:NoSchedule
    --node-labels=node.openshift.io/os_id=Windows --resolv-conf= --windows-priorityclass=ABOVE_NORMAL_PRIORITY_CLASS
    --v=2 --cloud-provider=external --hostname-override=HOSTNAME_OVERRIDE","powershellPreScripts":[{"variableName":"HOSTNAME_OVERRIDE","path":"Invoke-RestMethod
    -UseBasicParsing -Uri http://169.254.169.254/latest/meta-data/local-hostname"}],"dependencies":["containerd"],"bootstrap":true,"priority":1},{"name":"windows_exporter","path":"C:\\k\\windows_exporter.exe
    --collectors.enabled cpu,cs,logical_disk,net,os,service,system,textfile,container,memory,cpu_info","bootstrap":false,"priority":2},{"name":"hybrid-overlay-node","path":"C:\\k\\hybrid-overlay-node.exe
    --node NODE_NAME --bootstrap-kubeconfig=C:\\k\\kubeconfig --cert-dir=C:\\k\\cni\\config
    --cert-duration=24h --windows-service --logfile C:\\var\\log\\hybrid-overlay\\hybrid-overlay.log","nodeVariablesInCommand":[{"name":"NODE_NAME","nodeObjectJsonPath":"{.metadata.name}"}],"dependencies":["kubelet"],"bootstrap":false,"priority":2},{"name":"csi-proxy","path":"C:\\k\\csi-proxy.exe
    -log_file=C:\\var\\log\\csi-proxy\\csi-proxy.log -logtostderr=false -windows-service
    --v=2","bootstrap":false,"priority":2},{"name":"kube-proxy","path":"C:\\k\\kube-log-runner.exe
    -log-file=C:\\var\\log\\kube-proxy\\kube-proxy.log C:\\k\\kube-proxy.exe --windows-service
    --proxy-mode=kernelspace --feature-gates=WinOverlay=true --hostname-override=NODE_NAME
    --kubeconfig=C:\\k\\kubeconfig --cluster-cidr=NODE_SUBNET --network-name=OVNKubernetesHybridOverlayNetwork
    --source-vip=ENDPOINT_IP --enable-dsr=false --v=2","nodeVariablesInCommand":[{"name":"NODE_NAME","nodeObjectJsonPath":"{.metadata.name}"},{"name":"NODE_SUBNET","nodeObjectJsonPath":"{.metadata.annotations.k8s\\.ovn\\.org/hybrid-overlay-node-subnet}"}],"powershellPreScripts":[{"variableName":"ENDPOINT_IP","path":"C:\\Temp\\network-conf.ps1"}],"dependencies":["hybrid-overlay-node"],"bootstrap":false,"priority":3}]'
  watchedEnvironmentVars: '["HTTP_PROXY","HTTPS_PROXY","NO_PROXY"]'
immutable: true
kind: ConfigMap
metadata:
  creationTimestamp: "2023-11-09T19:04:35Z"
  name: windows-services-9.0.0-9456533
  namespace: winc-namespace-test
  resourceVersion: "188663"
  uid: 5e2d7221-2919-4f50-9778-32791dbf18d3


oc get rolebinding -n winc-namespace-test windows-instance-config-daemon -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  creationTimestamp: "2023-11-09T19:04:35Z"
  name: windows-instance-config-daemon
  namespace: winc-namespace-test
  resourceVersion: "188697"
  uid: b1fc1212-266f-42d5-b45f-801e3c74379f
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: windows-instance-config-daemon
subjects:
- kind: ServiceAccount
  name: windows-instance-config-daemon
  namespace: openshift-windows-machine-config-operator


oc get serviceaccount -n winc-namespace-test
NAME                              SECRETS   AGE
builder                           1         6m11s
default                           1         6m11s
deployer                          1         6m11s
windows-instance-config-daemon    1         6m3s
windows-machine-config-operator   1         6m2s