-
Bug
-
Resolution: Done-Errata
-
Normal
-
4.14
-
+
-
Critical
-
No
-
3
-
WINC - Sprint 238
-
1
-
False
-
-
-
Bug Fix
Description of problem:
Installing WMCO on AWS/Nutaix in a different namespace is failing, nodes are missing machines Stuck in provisioning state.
Version-Release number of selected component (if applicable):
4.14 9.0.0-c2b5660
How reproducible:
100%
Steps to Reproduce:
1.Installed WMCO in the default namespace 2. create a machineset of 2 replicas waited for them to be up and running 3. Create new namespace winc-namespace-test with openshift.io/cluster-monitoring: "True" 4. Scale down machineset to 0 5. delete the openshift-windows-machine-config-operator namesapace 6. Install the cloud-private-key 7. install the operator group 8. install the subscription 9. scale up windows machineset to 1 replica after the custom namespace created properly
Actual results:
Windows machines are not scaling up with the custom namespace number of Windwos nodes is 0
Expected results:
Scaling up with a new namespace
Additional info:
oc get rolebinding -n winc-namespace-test
NAME ROLE AGE
prometheus-k8s Role/prometheus-k8s 25m
system:deployers ClusterRole/system:deployer 26m
system:image-builders ClusterRole/system:image-builder 26m
system:image-pullers ClusterRole/system:image-puller 26m
windows-instance-config-daemon Role/windows-instance-config-daemon 25m
windows-machine-config-operator.v9.0.0 Role/windows-machine-config-operator.v9.0.0 25m
windows-machine-config-operator.v9.0.0-windows-machi-5c975c9cb9 Role/windows-machine-config-operator.v9.0.0-windows-machi-5c975c9cb9 25m
oc get roles -n winc-namespace-test
NAME CREATED AT
manager-role 2023-06-26T15:56:25Z
prometheus-k8s 2023-06-26T15:56:25Z
windows-instance-config-daemon 2023-06-26T15:56:25Z
windows-machine-config-operator.v9.0.0 2023-06-26T15:56:24Z
windows-machine-config-operator.v9.0.0-windows-machi-5c975c9cb9 2023-06-26T15:56:26Z
oc get rolebinding -n winc-namespace-test windows-machine-config-operator.v9.0.0 -oyaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: "2023-06-26T15:56:24Z"
name: windows-machine-config-operator.v9.0.0
namespace: winc-namespace-test
ownerReferences:
- apiVersion: operators.coreos.com/v2
blockOwnerDeletion: false
controller: true
kind: OperatorCondition
name: windows-machine-config-operator.v9.0.0
uid: 8cdd4afa-e8a2-4d25-993a-99191d3d735f
resourceVersion: "231139"
uid: 5e0df1c7-ba9f-48ee-ae3e-acf7043ae211
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: windows-machine-config-operator.v9.0.0
subjects:
- kind: ServiceAccount
name: windows-machine-config-operator
- kind: ServiceAccount
name: windows-machine-config-operatorRonnie 19 hours ago
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: "2023-06-26T15:56:24Z"
name: windows-machine-config-operator.v9.0.0
namespace: winc-namespace-test
ownerReferences:
- apiVersion: operators.coreos.com/v2
blockOwnerDeletion: false
controller: true
kind: OperatorCondition
name: windows-machine-config-operator.v9.0.0
uid: 8cdd4afa-e8a2-4d25-993a-99191d3d735f
resourceVersion: "231139"
uid: 5e0df1c7-ba9f-48ee-ae3e-acf7043ae211
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: windows-machine-config-operator.v9.0.0
subjects:
- kind: ServiceAccount
name: windows-machine-config-operator
- kind: ServiceAccount
name: windows-machine-config-operator
{"level":"info","ts":"2023-06-27T11:16:45Z","logger":"wc 10.0.140.19","msg":"configuring"}
{"level":"error","ts":"2023-06-27T11:16:48Z","logger":"wc 10.0.140.19","msg":"error running","cmd":"powershell.exe -NonInteractive -ExecutionPolicy Bypass \"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace winc-namespace-test\"","out":"F0627 11:16:48.322957 3596 cleanup.go:51] cannot get latest services ConfigMap from namespace winc-namespace-test: configmaps is forbidden: User \"system:serviceaccount:winc-namespace-test:windows-instance-config-daemon\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"winc-namespace-test\"\n","error":"Process exited with status 1","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Run\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:381\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).RunWICDCleanup\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:408\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:434\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:159\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:84\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:432\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226"}
{"level":"info","ts":"2023-06-27T11:16:48Z","logger":"wc 10.0.140.19","msg":"failed to cleanup node","command":"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace winc-namespace-test","output":"F0627 11:16:48.322957 3596 cleanup.go:51] cannot get latest services ConfigMap from namespace winc-namespace-test: configmaps is forbidden: User \"system:serviceaccount:winc-namespace-test:windows-instance-config-daemon\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"winc-namespace-test\"\n"}
- blocks
-
-
- Closed
-
- is cloned by
-
-
- Closed
-
- is related to
-
-
- Verified
-
- links to
- mentioned on
[OCPBUGS-15461] Cannot get latest services ConfigMap from custom namespace
Matthew Werner
added a comment - paravindh Can you please update the release note doc text for this bug fix? Thank you.
Matthew Werner
added a comment - paravindh Can you please update the release note doc text for this bug fix? Thank you. Verified in wmco-container-9.0.0-49 using two methods:
[jfrancoa@jfrancoa-thinkpadp1gen4i wmco]$ oc get cm -n winc-namespace-test
NAME DATA AGE
kube-root-ca.crt 1 11m
openshift-service-ca.crt 1 11m
windows-machine-config-operator-lock 0 8m28s
windows-services-9.0.0-979e746 2 11m
- Manually deletion and installation of WMCO in a different namespace. After installing WMCO in the winc-namespace-test namespace the rolebinding was pointing at the SA located in the new namespace:
[jfrancoa@jfrancoa-thinkpadp1gen4i 214587]$ oc scale machineset -n openshift-machine-api jfrancoa-0507-m6r7w-windows-worker-us-east-2a --replicas=0 machineset.machine.openshift.io/jfrancoa-0507-m6r7w-windows-worker-us-east-2a scaled [jfrancoa@jfrancoa-thinkpadp1gen4i 214587]$ cd ~/OpenShift/wmco/ [jfrancoa@jfrancoa-thinkpadp1gen4i wmco]$ oc delete -f subscription.yaml [252/629] subscription.operators.coreos.com "windows-machine-config-operator" deleted [jfrancoa@jfrancoa-thinkpadp1gen4i wmco]$ oc delete -f operatorgroup.yaml operatorgroup.operators.coreos.com "windows-machine-config-operator" deleted [jfrancoa@jfrancoa-thinkpadp1gen4i wmco]$ oc delete -f namespace.yaml namespace "openshift-windows-machine-config-operator" deleted [jfrancoa@jfrancoa-thinkpadp1gen4i wmco]$ cd manual_test/ [jfrancoa@jfrancoa-thinkpadp1gen4i manual_test]$ cd OCP37086/ [jfrancoa@jfrancoa-thinkpadp1gen4i OCP37086]$ ls catalogsource_local.yaml namespace.yaml operatorgroup.yaml subscription_qe.yaml subscription.yaml [jfrancoa@jfrancoa-thinkpadp1gen4i OCP37086]$ oc create -f namespace.yaml namespace/winc-namespace-test created [jfrancoa@jfrancoa-thinkpadp1gen4i OCP37086]$ oc create secret generic cloud-private-key --from-file=private-key.pem=/home/jfrancoa/.ssh/openshift-qe.pem -n winc-namespace-test secret/cloud-private-key created [jfrancoa@jfrancoa-thinkpadp1gen4i OCP37086]$ oc create -f operatorgroup.yaml operatorgroup.operators.coreos.com/windows-machine-config-operator created [jfrancoa@jfrancoa-thinkpadp1gen4i OCP37086]$ oc create -f subscription.yaml subscription.operators.coreos.com/windows-machine-config-operator created [jfrancoa@jfrancoa-thinkpadp1gen4i OCP37086]$ oc get csv -n winc-namespace-test NAME DISPLAY VERSION REPLACES PHASE windows-machine-config-operator.v9.0.0 Windows Machine Config Operator 9.0.0 Succeeded [jfrancoa@jfrancoa-thinkpadp1gen4i OCP37086]$ oc get rolebinding -n winc-namespace-test windows-instance-config-daemon -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: "2023-07-05T08:39:58Z" name: windows-instance-config-daemon namespace: winc-namespace-test resourceVersion: "86549" uid: 15dad070-5ba1-41e7-9f46-f77ac58a04f9 roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: windows-instance-config-daemon subjects: - kind: ServiceAccount name: windows-instance-config-daemon namespace: winc-namespace-test [jfrancoa@jfrancoa-thinkpadp1gen4i OCP37086]$ oc scale machineset -n openshift-machine-api jfrancoa-0507-m6r7w-windows-wo rker-us-east-2a --replicas=2 [jfrancoa@jfrancoa-thinkpadp1gen4i OCP37086]$ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-133-52.us-east-2.compute.internal Ready control-plane,master 3h50m v1.27.3+ab0b8ee ip-10-0-150-176.us-east-2.compute.internal Ready worker 3h40m v1.27.3+ab0b8ee ip-10-0-152-92.us-east-2.compute.internal Ready worker 31m v1.27.3+cb4b47e ip-10-0-154-247.us-east-2.compute.internal Ready worker 35m v1.27.3+cb4b47e ip-10-0-160-97.us-east-2.compute.internal Ready worker 3h39m v1.27.3+ab0b8ee ip-10-0-184-253.us-east-2.compute.internal Ready control-plane,master 3h50m v1.27.3+ab0b8ee ip-10-0-207-176.us-east-2.compute.internal Ready control-plane,master 3h51m v1.27.3+ab0b8ee ip-10-0-211-8.us-east-2.compute.internal Ready worker 3h41m v1.27.3+ab0b8ee
- Then, restored the cluster back installing WMCO in the default namespace. Opened the CLI WebConsole and deleted the WMCO installation from there, followed by installing it on the winc-namespace-test namespace. The rolebinding was pointing at the right namespace and the nodes could scale up:
[jfrancoa@jfrancoa-thinkpadp1gen4i wmco]$ oc get rolebinding -n winc-namespace-test windows-instance-config-daemon -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: "2023-07-05T10:41:54Z" name: windows-instance-config-daemon namespace: winc-namespace-test resourceVersion: "130786" uid: 5a651297-8941-4945-b11e-a60e055715b7 roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: windows-instance-config-daemon subjects: - kind: ServiceAccount name: windows-instance-config-daemon namespace: winc-namespace-test [jfrancoa@jfrancoa-thinkpadp1gen4i wmco]$ oc scale machineset -n openshift-machine-api jfrancoa-0507-m6r7w-windows-worker-us-east-2a --replicas=2 machineset.machine.openshift.io/jfrancoa-0507-m6r7w-windows-worker-us-east-2a scaled
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-winc-midstream / openshift-winc-midstream on branch rhaos-4.14-rhel-9_upstream_a3befb16fcf2d9d477dda9d1515980eb:
Updated US source to: 979e746 Merge pull request #1671 from aravindhp/
OCPBUGS-15461
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in a merge request of openshift-winc-midstream / openshift-winc-midstream on branch rhaos-4.14-rhel-9_ upstream _a3befb16fcf2d9d477dda9d1515980eb : Updated US source to: 979e746 Merge pull request #1671 from aravindhp/ OCPBUGS-15461 The bug is in ensureWICDClusterRoleBinding() where we are not checking if the namespace has changed to decided whether to delete the WICD ClusterRoleBinding.
I could confirm that the same thing occurs using the OpenShift Console (web GUI).
Having WMCO installed on openshift-windows-machine-config-operator with two Windows Machine nodes I scaled down the MachineSet from 2 to 0 and then went to the console.
There I uninstalled WMCO clicking in Installed Operators -> (Three dots in WMCO) Uninstall
I waited couple of minutes and went to the OperatorHub. There I looked for WMCO and installed it. In namespace I specified winc-namespace-test. Then I went to the cluster and I could see the rolebinding pointing at openshift-windows-machine-config-operator:
[jfrancoa@localhost OCP37086]$ oc get pods -n winc-namespace-test NAME READY STATUS RESTARTS AGE windows-machine-config-operator-6f6c985876-k8t25 1/1 Running 0 25s [jfrancoa@localhost OCP37086]$ oc get rolebinding -n openshift-windows-machine-config-operator windows-instance-config-daemon -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: "2023-06-28T11:54:58Z" name: windows-instance-config-daemon namespace: openshift-windows-machine-config-operator resourceVersion: "150522" uid: ccefaf14-c81b-47f3-9e7b-fb211476155d roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: windows-instance-config-daemon subjects: - kind: ServiceAccount name: windows-instance-config-daemon namespace: openshift-windows-machine-config-operator [jfrancoa@localhost OCP37086]$ oc get serviceaccount -n winc-namespace-test NAME SECRETS AGE builder 1 82s default 1 82s deployer 1 81s windows-instance-config-daemon 1 74s windows-machine-config-operator 1 73s [jfrancoa@localhost OCP37086]$ oc get all -n openshift-windows-machine-config-operator No resources found in openshift-windows-machine-config-operator namespace.
On the other side, it looks that when performing a WMCO restart the problem gets solved:
[jfrancoa@localhost OCP37086]$ oc logs deployment/windows-machine-config-operator -n winc-namespace-test -f {"level":"info","ts":"2023-06-28T12:22:19Z","logger":"version","msg":"operator","version":"9.0.0-f079f3d"} {"level":"info","ts":"2023-06-28T12:22:19Z","logger":"version","msg":"go","version":"go1.20.4 linux/amd64"} {"level":"info","ts":"2023-06-28T12:22:19Z","logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":"2023-06-28T12:22:19Z","logger":"leader","msg":"No pre-existing lock was found."} .................................................................................................................................................................... {"level":"error","ts":"2023-06-28T12:40:04Z","logger":"wc 10.0.153.246","msg":"error running","cmd":"powershell.exe -NonInteractive -ExecutionPolicy Bypass \"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace winc-namespace-test\"","out":"I0628 12:40:04.505998 1316 cleanup.go:53] no associated node found\nF0628 12:40:04.560396 1316 cleanup.go:51] cannot get latest services ConfigMap from namespace winc-namespace-test: configmaps is forbidden: User \"system:serviceaccount:winc-namespace-test:windows-instance-config-daemon\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"winc-namespace-test\"\n","error":"Process exited with status 1","stacktrace":"github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Run\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:381\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).RunWICDCleanup\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:408\ngithub.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap\n\t/remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:434\ngithub.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure\n\t/remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:159\ngithub.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate\n\t/remote-source/build/windows-machine-config-operator/controllers/controllers.go:84\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:432\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226"} {"level":"info","ts":"2023-06-28T12:40:04Z","logger":"wc 10.0.153.246","msg":"failed to cleanup node","command":"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace winc-namespace-test","output":"I0628 12:40:04.505998 1316 cleanup.go:53] no associated node found\nF0628 12:40:04.560396 1316 cleanup.go:51] cannot get latest services ConfigMap from namespace winc-namespace-test: configmaps is forbidden: User \"system:serviceaccount:winc-namespace-test:windows-instance-config-daemon\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"winc-namespace-test\"\n"} {"level":"error","ts":"2023-06-28T12:40:04Z","msg":"Reconciler error","controller":"machine","controllerGroup":"machine.openshift.io","controllerKind":"Machine","Machine":{"name":"jfrancoa-aws-2806-blwcr-windows-worker-us-east-2a-cxqqk","namespace":"openshift-machine-api"},"namespace":"openshift-machine-api","name":"jfrancoa-aws-2806-blwcr-windows-worker-us-east-2a-cxqqk","reconcileID":"2c8f15d2-4423-47fe-bb8f-38ff7787b0a2","error":"unable to configure instance i-0e9f02f3487accc3c: bootstrapping the Windows instance failed: unable to cleanup the Windows instance: error running powershell.exe -NonInteractive -ExecutionPolicy Bypass \"C:\\k\\windows-instance-config-daemon.exe cleanup --kubeconfig C:\\k\\wicd-kubeconfig --namespace winc-namespace-test\": Process exited with status 1","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226"} {"level":"info","ts":"2023-06-28T12:40:04Z","logger":"controller.windowsmachine","msg":"processing","windowsmachine":{"Name":"jfrancoa-aws-2806-blwcr-windows-worker-us-east-2a-cxqqk","Namespace":"openshift-machine-api"},"address":"10.0.153.246"} [jfrancoa@localhost OCP37086]$ oc get pods -n winc-namespace-test NAME READY STATUS RESTARTS AGE windows-machine-config-operator-6f6c985876-k8t25 1/1 Running 0 18m [jfrancoa@localhost OCP37086]$ oc delete pod windows-machine-config-operator-6f6c985876-k8t25 -n winc-namespace-test pod "windows-machine-config-operator-6f6c985876-k8t25" deleted [jfrancoa@localhost OCP37086]$ oc get rolebinding -n winc-namespace-test windows-instance-config-daemon -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: "2023-06-28T12:41:06Z" name: windows-instance-config-daemon namespace: winc-namespace-test resourceVersion: "168415" uid: 63587bc1-3b86-453d-8dff-63b089adbf2f roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: windows-instance-config-daemon subjects: - kind: ServiceAccount name: windows-instance-config-daemon namespace: winc-namespace-test <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<----RIGHT VALUE
I managed to reproduce it manually. The steps done were:
- Deploy IPI on AWS with WMCO 9.0.0 (windows-services-9.0.0-f079f3d) on the default namespace openshift-windows-machine-config-operator
- Create a Windows MachineSet (Windows 2019), scale it up to two and wait for the nodes to be in Ready state
- Create workloads that will land in those two Windows Machine nodes
- Scale down the Windows MachineSet to 0
- Uninstall WMCO by: deleting the subscription, the operator group and the namespace
- Install back WMCO in winc-namespace-test: create a winc-namespace-test, create the WMCO secret on the namespace, create operator group and subscription
- Wait for WMCO to be up and running
- Scale up the Windows MachineSet back to 2
- Wait for the nodes to be in ready state. But the nodes never come up and the log Ronnie mentioned originally appears.
paravindh to delete WMCO installation is there anything else missing? I though removing the subscription, then the operator group and the namespace was enough...but we might be missing something else.
When checking the windows-instance-config-daemon rolebinding, it is still pointing at the service account windows-instance-config-daemon:openshift-windows-machine-config-operator:
[jfrancoa@localhost OCP37086]$ oc get rolebinding -n winc-namespace-test windows-instance-config-daemon -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: creationTimestamp: "2023-06-28T08:48:35Z" name: windows-instance-config-daemon namespace: winc-namespace-test resourceVersion: "82361" uid: 9f975bcb-4edb-4267-8b88-35cd42670f1b roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: windows-instance-config-daemon subjects: - kind: ServiceAccount name: windows-instance-config-daemon namespace: openshift-windows-machine-config-operator
But no serviceaccount is present there, only on the winc-namespace-test:
[jfrancoa@localhost OCP37086]$ oc get serviceaccount -n winc-namespace-test NAME SECRETS AGE builder 1 37m default 1 37m deployer 1 37m windows-instance-config-daemon 1 24m windows-machine-config-operator 1 24m [jfrancoa@localhost OCP37086]$ oc get serviceaccount -n openshift-windows-machine-config-operator No resources found in openshift-windows-machine-config-operator namespace.
I tried reproducing the bug with the following steps:
- Built a WMCO image from f079f3d
- Installed WMCO in the default namespace, openshift-windows-machine-config-operator, by running hack/olm.sh run -k ~/.ssh/openshift-dev.pem
- Brought up a MachineSet with 2 replicas
- Waited for the nodes to be configured
- Uninstalled WMCO in the default namespace by running hack/olm.sh cleanup
- Scaled down the MachineSet to 0 replicas
- Installed WMCO in the non-default namespace by running export WMCO_DEPLOY_NAMESPACE=wmco-alternate; hack/olm.sh run -k ~/.ssh/openshift-dev.pem
- Scaled the MachineSet to 1 replica
- The node came up sucessfully
IOW, I was unable to reproduce the bug. I am attaching the logs for run: wmco-default-namespace.log
wmco-alternate-namespace.log
rrasouli I suspect something is not getting cleaned up correctly when in your uninstall WMCO in the default namespace step. Please look into that. Are you deleting the default WMCO namespace?
Your log has the following line:
{"level":"info","ts":"2023-06-27T11:03:09Z","logger":"version","msg":"operator","version":"9.0.0-c2b5660"}
This maps to commit c2b5660 which does not include the fix, OCPBUGS-13780,OCPBUGS-14700: Use latest and version CM for cleanup.. Please try again with a newer WMCO image: windows-machine-config-operator-container-9.0.0-46
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Important: Red Hat OpenShift for Windows Containers 9.0.0 security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2023:7515