-
Bug
-
Resolution: Done
-
Critical
-
4.12
-
None
-
None
-
0
-
WINC - Sprint 230
-
1
-
Rejected
-
False
-
This is a clone of issue OCPBUGS-5732. The following is the description of the original issue:
—
Description of problem:
During the validation of [OCPBUGS-4247|https://issues.redhat.com/browse/OCPBUGS-4247] I observed that all the workloads remained in the same Windows worker node during the whole upgrade process, however if an upgrade is taking place the kubelet will be impacted by such an upgrade so the workloads need to be move away from that node before the reconfiguration of the node occurss. This was confirmed by Mohammad: _I think I found the cause btw. For machine nodes, we don't try to find an instance associated with the machine being reconciled, instead just initializing the instanceInfo with a nil node. So when we check if an upgrade is required (i.e. should we deconfigure), we get false_ This behavior was included as part of the bug: [OCPBUGS-3506|https://issues.redhat.com/browse/OCPBUGS-3506] , which got cherry-picked into 4.12 and 4.11 too, therefore this bug impacts all those versions. Adding wmco logs as well as the traces which confirm that none of the workloads are moving out from the nodes to be reconfigured.
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Deploy a IPI cluster with Windows workers. Create some workloads for those Windows workers
2. Perform and upgrade or simply modify the version annotation of each of the worker nodes
3. Wait for WMCO to reconfigure (or upgrade) all the windows workers. Keep track on where those workers are landing, yo can use the following snippet for it:
lb=`oc get svc -l app=win-webserver -n winc-test -o=jsonpath="{.items[0].status.loadBalancer.ingress[0].hostname}"`;file=/tmp/35707_AWS_412.log;for i in {1..60}; do time=`date`; echo -e "\n#######ATTEMTP #${i} ${time} ######" &>> $file;oc get nodes -l=node.openshift.io/os_id="Windows" &>> $file;oc get pods -n winc-test -o wide &>> $file;curl --connect-timeout 60 $lb &>> $file;sleep 60; done
Actual results:
None of the Windows nodes get drained during the upgrade. Workloads remain in the same node which got reconfigured.
Expected results:
The Windows nodes get drained during the upgrade, right before WMCO reconfigures them.
Additional info:
- blocks
-
-
- Closed
-
- clones
-
-
- Closed
-
- is blocked by
-
-
- Closed
-
- is cloned by
-
-
- Closed
-
- links to
- mentioned on
(1 mentioned on)
[OCPBUGS-5803] Windows nodes do not get drained (deconfigure) during the upgrade process
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory, and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2022:9096
Verified in GCP IPI 4.12, using the version annotation test case and a helper command to get data on what workloads are doing during the upgrade.
Version:
$ oc get cm -n openshift-windows-machine-config-operator NAME DATA AGE kube-root-ca.crt 1 111m openshift-service-ca.crt 1 111m windows-machine-config-operator-lock 0 37m windows-services-7.0.0-c9c49be 2 110m $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.0 True False 116m Cluster version is 4.12.0
We can now see, that right before the worker is about to be reconfigured all the workloads move away from that node, therefore the node is drained before reconfiguring it:
#### INITIAL STATE #### #######ATTEMTP #13 Tue Jan 17 06:10:50 EST 2023 ###### NAME STATUS ROLES AGE VERSION jfrancoa-1701-4ng9q-windows-worker-a-cf7dk Ready worker 60m v1.25.0-2653+a34b9e9499e6c3 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 Ready worker 63m v1.25.0-2653+a34b9e9499e6c3 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 Ready worker 5m21s v1.25.0-2653+a34b9e9499e6c3 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES linux-webserver-8fc5c86b8-c59j7 1/1 Running 0 59m 10.131.0.25 jfrancoa-1701-4ng9q-worker-b-x8gkk.c.openshift-qe.internal <none> <none> win-webserver-745df6565f-28ls6 1/1 Running 0 3m20s 10.132.1.3 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-429kx 1/1 Running 0 3m20s 10.132.2.2 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-hfspc 1/1 Running 0 60m 10.132.0.5 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-jtgcf 1/1 Running 0 3m20s 10.132.2.3 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-mzkh2 1/1 Running 0 60m 10.132.0.3 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-nctpg 1/1 Running 0 3m20s 10.132.1.2 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-rz7hk 1/1 Running 0 60m 10.132.0.2 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-twfsm 1/1 Running 0 60m 10.132.0.6 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-wtm54 1/1 Running 0 60m 10.132.0.4 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> #### UPGRADE OF NODE jfrancoa-1701-4ng9q-windows-worker-a-cf7dk (No workloads found in that node)##### #######ATTEMTP #14 Tue Jan 17 06:11:50 EST 2023 ###### NAME STATUS ROLES AGE VERSION jfrancoa-1701-4ng9q-windows-worker-a-cf7dk Ready,SchedulingDisabled worker 61m v1.25.0-2653+a34b9e9499e6c3 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 Ready worker 64m v1.25.0-2653+a34b9e9499e6c3 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 Ready worker 6m22s v1.25.0-2653+a34b9e9499e6c3 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES linux-webserver-8fc5c86b8-c59j7 1/1 Running 0 60m 10.131.0.25 jfrancoa-1701-4ng9q-worker-b-x8gkk.c.openshift-qe.internal <none> <none> win-webserver-745df6565f-429kx 1/1 Running 0 4m20s 10.132.2.2 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-hfspc 1/1 Running 0 61m 10.132.0.5 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-jtgcf 1/1 Running 0 4m20s 10.132.2.3 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-mzkh2 1/1 Running 0 61m 10.132.0.3 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-nw86x 1/1 Running 0 54s 10.132.2.5 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-r9t9b 1/1 Running 0 54s 10.132.2.4 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-rz7hk 1/1 Running 0 61m 10.132.0.2 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-twfsm 1/1 Running 0 61m 10.132.0.6 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-wtm54 1/1 Running 0 61m 10.132.0.4 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> #### UPGRADE OF NODE jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 (No workloads found in that node)##### #######ATTEMTP #22 Tue Jan 17 06:19:54 EST 2023 ###### NAME STATUS ROLES AGE VERSION jfrancoa-1701-4ng9q-windows-worker-a-cf7dk Ready worker 69m v1.25.0-2653+a34b9e9499e6c3 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 Ready worker 72m v1.25.0-2653+a34b9e9499e6c3 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 Ready,SchedulingDisabled worker 14m v1.25.0-2653+a34b9e9499e6c3 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES linux-webserver-8fc5c86b8-c59j7 1/1 Running 0 68m 10.131.0.25 jfrancoa-1701-4ng9q-worker-b-x8gkk.c.openshift-qe.internal <none> <none> win-webserver-745df6565f-77j8x 1/1 Running 0 39s 10.132.1.7 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-bwk7f 1/1 Running 0 39s 10.132.1.6 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-hfspc 1/1 Running 0 69m 10.132.0.5 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-ltk8w 1/1 Running 0 39s 10.132.1.4 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-mzkh2 1/1 Running 0 69m 10.132.0.3 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-n7kk9 1/1 Running 0 39s 10.132.1.5 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-rz7hk 1/1 Running 0 69m 10.132.0.2 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-twfsm 1/1 Running 0 69m 10.132.0.6 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> win-webserver-745df6565f-wtm54 1/1 Running 0 69m 10.132.0.4 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 <none> <none> #### UPGRADE OF NODE jfrancoa-1701-4ng9q-windows-worker-a-w46z8 (No workloads found in that node)##### #######ATTEMTP #30 Tue Jan 17 06:27:58 EST 2023 ###### NAME STATUS ROLES AGE VERSION jfrancoa-1701-4ng9q-windows-worker-a-cf7dk Ready worker 77m v1.25.0-2653+a34b9e9499e6c3 jfrancoa-1701-4ng9q-windows-worker-a-w46z8 Ready,SchedulingDisabled worker 80m v1.25.0-2653+a34b9e9499e6c3 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 Ready worker 22m v1.25.0-2653+a34b9e9499e6c3 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES linux-webserver-8fc5c86b8-c59j7 1/1 Running 0 76m 10.131.0.25 jfrancoa-1701-4ng9q-worker-b-x8gkk.c.openshift-qe.internal <none> <none> win-webserver-745df6565f-2pjvc 1/1 Running 0 19s 10.132.2.6 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-77j8x 1/1 Running 0 8m43s 10.132.1.7 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-bwk7f 1/1 Running 0 8m43s 10.132.1.6 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-f4sdb 1/1 Running 0 19s 10.132.2.8 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-hcrc2 1/1 Running 0 19s 10.132.2.9 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-ltk8w 1/1 Running 0 8m43s 10.132.1.4 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-n7kk9 1/1 Running 0 8m43s 10.132.1.5 jfrancoa-1701-4ng9q-windows-worker-a-cf7dk <none> <none> win-webserver-745df6565f-p29b8 1/1 Running 0 19s 10.132.2.10 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none> win-webserver-745df6565f-vjpqn 1/1 Running 0 19s 10.132.2.7 jfrancoa-1701-4ng9q-windows-worker-a-zrfj2 <none> <none>
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in merge request !93 of openshift-winc-midstream / openshift-winc-midstream on branch rhaos-4.12-rhel-8_upstream_cb63a1e276bd3db378b411af7040fbf2:
Updated US source to: 1853b0d Merge pull request #1382 from openshift-cherrypick-robot/cherry-pick-1378-to-release-4.12
GitLab CEE Bot
added a comment - CPaaS Service Account mentioned this issue in merge request !93 of openshift-winc-midstream / openshift-winc-midstream on branch rhaos-4.12-rhel-8_ upstream _cb63a1e276bd3db378b411af7040fbf2 : Updated US source to: 1853b0d Merge pull request #1382 from openshift-cherrypick-robot/cherry-pick-1378-to-release-4.12
Per the announcement sent regarding the removal of "Blocker" as an option in the Priority field, this issue (which was already closed at the time of the bulk update) had Priority = "Blocker." It is being updated to Priority = Critical. No additional fields were changed.