-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
4.13.z
-
Moderate
-
No
-
False
-
-
Release Note Not Required
-
In Progress
Description of problem:
When a new configuration is picked up by the authentication operator, it rolls out a new revision of oauth-server pods. However, since the pods define `terminationGracePeriodSeconds`, the old-revision pods would still be running even after the oauth-server deployment reports that all pods have been updated to the newest revision, which triggers the authentication operator to report Progressing=false. The above behavior might cause that tests (and possible any observers) that expect the newer revision would be still routed to the old pods, causing confusion.
Version-Release number of selected component (if applicable):
4.13
How reproducible:
Always
Steps to Reproduce:
1. Trigger new oauth-server rollout 2. Observe the authentication operator reporting Progressing while also watching the number of pods in the openshift-authentication namespace
Actual results:
CAO reports Progressing=false even though there are more than the expected number of pods running.
Expected results:
CAO waits to report Progressing=false when only the new revision of pods is running in the openshift-authentication NS.
Additional info:
-
- is cloned by
-
-
- Verified
-
- is depended on by
-
-
- Verified
-
- relates to
-
-
- New
-
- links to
-
RHEA-2024:11038
OpenShift Container Platform 4.19.z bug fix update
(3 links to)
[OCPBUGS-23435] authn op. reports Progressing=false when old-gen pods are still running
rhn-support-rgangwar , from your output, all old pods and new pods are apiserver-54c4bfbcb4-xxxx, i.e. they are same generation, no new generation? And from AGE 87m and 88m, not all three pods go through rollout?
Rahul Gangwar
added a comment - Not see issue on OAS, during oas pods terminating and initialising pods. OAS is in progressing=True
1. Trigger openshift-apiserver to roll out while true; do date; oc get co openshift-apiserver; oc get po -n openshift-apiserver -o wide; echo; done I see the output: Thu Nov 28 17:51:39 IST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True False False 99m NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES apiserver-54c4bfbcb4-hkrld 2/2 Running 0 85m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> apiserver-54c4bfbcb4-n4vkw 2/2 Running 0 82m 10.130.0.88 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none> apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 87m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none> Thu Nov 28 17:51:41 IST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True True False 99m APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: observed generation is 8, desired generation is 10. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES apiserver-54c4bfbcb4-gxvvd 0/2 Pending 0 1s <none> <none> <none> <none> apiserver-54c4bfbcb4-hkrld 2/2 Running 0 85m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> apiserver-54c4bfbcb4-n4vkw 2/2 Terminating 0 83m 10.130.0.88 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none> apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 87m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none> Thu Nov 28 17:53:30 IST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True True False 101m APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 3/3 pods have been updated to the latest generation and 2/3 pods are available NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES apiserver-54c4bfbcb4-gxvvd 0/2 Pending 0 110s <none> <none> <none> <none> apiserver-54c4bfbcb4-hkrld 2/2 Running 0 86m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> apiserver-54c4bfbcb4-n4vkw 1/2 Terminating 0 84m 10.130.0.88 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none> apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 88m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none> Thu Nov 28 17:53:33 IST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True True False 101m APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 3/3 pods have been updated to the latest generation and 2/3 pods are available NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES apiserver-54c4bfbcb4-gxvvd 0/2 PodInitializing 0 112s 10.130.0.92 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none> apiserver-54c4bfbcb4-hkrld 2/2 Running 0 86m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 88m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none> Thu Nov 28 17:53:35 IST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True True False 101m APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 3/3 pods have been updated to the latest generation and 2/3 pods are available NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES apiserver-54c4bfbcb4-gxvvd 1/2 Running 0 115s 10.130.0.92 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none> apiserver-54c4bfbcb4-hkrld 2/2 Running 0 87m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 88m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <non
Moving this bug to verified
Rahul Gangwar
added a comment - Not see issue on OAS, during oas pods terminating and initialising pods. OAS is in progressing=True
1. Trigger openshift-apiserver to roll out
while true ; do date; oc get co openshift-apiserver; oc get po -n openshift-apiserver -o wide; echo; done
I see the output:
Thu Nov 28 17:51:39 IST 2024
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True False False 99m
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apiserver-54c4bfbcb4-hkrld 2/2 Running 0 85m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none>
apiserver-54c4bfbcb4-n4vkw 2/2 Running 0 82m 10.130.0.88 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none>
apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 87m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none>
Thu Nov 28 17:51:41 IST 2024
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True True False 99m APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: observed generation is 8, desired generation is 10.
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apiserver-54c4bfbcb4-gxvvd 0/2 Pending 0 1s <none> <none> <none> <none>
apiserver-54c4bfbcb4-hkrld 2/2 Running 0 85m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none>
apiserver-54c4bfbcb4-n4vkw 2/2 Terminating 0 83m 10.130.0.88 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none>
apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 87m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none>
Thu Nov 28 17:53:30 IST 2024
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True True False 101m APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 3/3 pods have been updated to the latest generation and 2/3 pods are available
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apiserver-54c4bfbcb4-gxvvd 0/2 Pending 0 110s <none> <none> <none> <none>
apiserver-54c4bfbcb4-hkrld 2/2 Running 0 86m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none>
apiserver-54c4bfbcb4-n4vkw 1/2 Terminating 0 84m 10.130.0.88 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none>
apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 88m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none>
Thu Nov 28 17:53:33 IST 2024
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True True False 101m APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 3/3 pods have been updated to the latest generation and 2/3 pods are available
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apiserver-54c4bfbcb4-gxvvd 0/2 PodInitializing 0 112s 10.130.0.92 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none>
apiserver-54c4bfbcb4-hkrld 2/2 Running 0 86m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none>
apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 88m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none>
Thu Nov 28 17:53:35 IST 2024
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
openshift-apiserver 4.19.0-0.nightly-2024-11-28-055536 True True False 101m APIServerDeploymentProgressing: deployment/apiserver.openshift-apiserver: 3/3 pods have been updated to the latest generation and 2/3 pods are available
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
apiserver-54c4bfbcb4-gxvvd 1/2 Running 0 115s 10.130.0.92 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none>
apiserver-54c4bfbcb4-hkrld 2/2 Running 0 87m 10.129.0.70 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none>
apiserver-54c4bfbcb4-z6tzz 2/2 Running 0 88m 10.128.0.75 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <non
Moving this bug to verified Once rhn-support-rgangwar is done in verifying OAS-O, he will help move the status to Verified.
Launched 4.19.0-0.nightly-2024-11-28-055536 env, ran above reproducer steps three times, none hit the issue again. Below is output of one trial:
Thu Nov 28 07:22:45 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.19.0-0.nightly-2024-11-28-055536 True True False 24m OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: 1/3 pods have been updated to the latest generation and 2/3 pods are available NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-569c4d8868-77cc9 1/1 Running 0 60s 10.129.0.77 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> oauth-openshift-569c4d8868-867mn 0/1 Pending 0 7s <none> <none> <none> <none> oauth-openshift-569c4d8868-kgcb2 1/1 Running 0 34s 10.130.0.91 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none> oauth-openshift-76f9c456b8-nkrpg 1/1 Terminating 0 2m45s 10.128.0.87 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none> ... ... Thu Nov 28 07:23:41 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.19.0-0.nightly-2024-11-28-055536 True True False 25m OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: 3/3 pods have been updated to the latest generation and 2/3 pods are available NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-569c4d8868-77cc9 1/1 Running 0 77s 10.129.0.77 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> oauth-openshift-569c4d8868-867mn 0/1 Pending 0 24s <none> <none> <none> <none> oauth-openshift-569c4d8868-kgcb2 1/1 Running 0 51s 10.130.0.91 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none> oauth-openshift-76f9c456b8-nkrpg 1/1 Terminating 0 3m2s 10.128.0.87 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none> Thu Nov 28 07:23:43 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.19.0-0.nightly-2024-11-28-055536 True True False 25m OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: 3/3 pods have been updated to the latest generation and 2/3 pods are available NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-569c4d8868-77cc9 1/1 Running 0 79s 10.129.0.77 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> oauth-openshift-569c4d8868-867mn 0/1 ContainerCreating 0 26s <none> ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none> oauth-openshift-569c4d8868-kgcb2 1/1 Running 0 53s 10.130.0.91 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none> Thu Nov 28 07:23:45 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.19.0-0.nightly-2024-11-28-055536 True True False 25m OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: 3/3 pods have been updated to the latest generation and 2/3 pods are available NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-569c4d8868-77cc9 1/1 Running 0 81s 10.129.0.77 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> oauth-openshift-569c4d8868-867mn 1/1 Running 0 28s 10.128.0.89 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none> oauth-openshift-569c4d8868-kgcb2 1/1 Running 0 55s 10.130.0.91 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none> Thu Nov 28 07:23:47 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.19.0-0.nightly-2024-11-28-055536 True False False 25m NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-569c4d8868-77cc9 1/1 Running 0 84s 10.129.0.77 ci-ln-cs2fi02-1d09d-tnd9d-master-0 <none> <none> oauth-openshift-569c4d8868-867mn 1/1 Running 0 31s 10.128.0.89 ci-ln-cs2fi02-1d09d-tnd9d-master-2 <none> <none> oauth-openshift-569c4d8868-kgcb2 1/1 Running 0 58s 10.130.0.91 ci-ln-cs2fi02-1d09d-tnd9d-master-1 <none> <none>
At Nov 28 07:22:45 PM CST 2024, all new pods are already shown, one old-gen pod oauth-openshift-76f9c456b8-nkrpg is still running (i.e. Terminating) until Nov 28 07:23:43 PM CST 2024. During that time window, the Progressing keeps being True, which is the fix's goal. So the issue is fixed.
Currently I do not have a 4.19 env on hand, will verify using above reproducer script once my env is up later.
fkrepins@redhat.com , sure, will verify backport too. One thing, it was my ignorance: as early as 2021, I observed this issue in automation tests https://github.com/openshift/cucushift/pull/8378 where the co/authentication transitioned to Progressing=False while the old-gen pods were still Terminating, but I did not consider it a bug, because I thought it was minor, no harm, just needed a few more seconds waiting time; thus, instead, I fixed our auto to wait more time for the old-gen pods to disappear :face-palm:
Today, first, again, I tried below to double manually reproduce it in unfixed version (4.17.6) as below, CC rhn-support-rgangwar :
1. Trigger oauth-openshift to roll out 2. Immediately run: $ while true; do date; oc get co authentication; oc get po -n openshift-authentication -o wide; echo; done I see the output: ... ... Thu Nov 28 05:00:03 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.17.6 True True False 69m OAuthServerDeploymentProgressing: deployment/oauth-openshift.openshift-authentication: 2/3 pods have been updated to the latest generation NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-76b74b5855-75hds 1/1 Running 0 62s 10.128.0.97 lhy-417-1128-1-xzb85-master-0 <none> <none> oauth-openshift-76b74b5855-qkhm2 0/1 Pending 0 6s <none> <none> <none> <none> oauth-openshift-76b74b5855-tmxvv 1/1 Running 0 34s 10.129.0.78 lhy-417-1128-1-xzb85-master-1 <none> <none> oauth-openshift-845867d688-sx55h 1/1 Terminating 0 13m 10.130.0.90 lhy-417-1128-1-xzb85-master-2 <none> <none> Thu Nov 28 05:00:04 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.17.6 True False False 69m NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-76b74b5855-75hds 1/1 Running 0 63s 10.128.0.97 lhy-417-1128-1-xzb85-master-0 <none> <none> oauth-openshift-76b74b5855-qkhm2 0/1 Pending 0 7s <none> <none> <none> <none> oauth-openshift-76b74b5855-tmxvv 1/1 Running 0 35s 10.129.0.78 lhy-417-1128-1-xzb85-master-1 <none> <none> oauth-openshift-845867d688-sx55h 1/1 Terminating 0 13m 10.130.0.90 lhy-417-1128-1-xzb85-master-2 <none> <none> ... ... Thu Nov 28 05:00:21 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.17.6 True False False 69m NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-76b74b5855-75hds 1/1 Running 0 81s 10.128.0.97 lhy-417-1128-1-xzb85-master-0 <none> <none> oauth-openshift-76b74b5855-qkhm2 0/1 Pending 0 25s <none> <none> <none> <none> oauth-openshift-76b74b5855-tmxvv 1/1 Running 0 53s 10.129.0.78 lhy-417-1128-1-xzb85-master-1 <none> <none> oauth-openshift-845867d688-sx55h 1/1 Terminating 0 13m 10.130.0.90 lhy-417-1128-1-xzb85-master-2 <none> <none> Thu Nov 28 05:00:23 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.17.6 True False False 69m NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-76b74b5855-75hds 1/1 Running 0 82s 10.128.0.97 lhy-417-1128-1-xzb85-master-0 <none> <none> oauth-openshift-76b74b5855-qkhm2 0/1 ContainerCreating 0 26s <none> lhy-417-1128-1-xzb85-master-2 <none> <none> oauth-openshift-76b74b5855-tmxvv 1/1 Running 0 54s 10.129.0.78 lhy-417-1128-1-xzb85-master-1 <none> <none> Thu Nov 28 05:00:24 PM CST 2024 NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.17.6 True False False 69m NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES oauth-openshift-76b74b5855-75hds 1/1 Running 0 84s 10.128.0.97 lhy-417-1128-1-xzb85-master-0 <none> <none> oauth-openshift-76b74b5855-qkhm2 1/1 Running 0 28s 10.130.0.94 lhy-417-1128-1-xzb85-master-2 <none> <none> oauth-openshift-76b74b5855-tmxvv 1/1 Running 0 56s 10.129.0.78 lhy-417-1128-1-xzb85-master-1 <none> <none> ... ...
i.e. at Nov 28 05:00:04 PM CST 2024, Progressing already turned False, but it should not, because the old-gen pod oauth-openshift-845867d688-sx55h was still running (i.e. Terminating) until Nov 28 05:00:21 PM CST 2024.
xxia-1 rhn-support-rgangwar Thanks for looking into it. Btw there is also a backport to 4.18 https://issues.redhat.com/browse/OCPBUGS-45051
Rahul Gangwar
added a comment - xxia-1 Sure I will do it rhn-support-rgangwar , OAS-O PR is also involved, I'll verify Auth PR, meantime would you like to help check OAS-O PR and if verify it if OAS-O had same issue? Thx! (Not urgent, just messaging first)
xxia-1 Updated, all pods got rollout