-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
4.10.z
-
None
-
Rejected
-
False
-
Description of problem:
While running loaded upgrade from 4.10.z to 4.11.z release on a 20 worker node vSphere cluster, noticed that the cluster operators are reported as degraded after upgrading cluster from 4.10.39 to 4.11.12. #oc get co:NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE 11-01 18:30:57.915 console 4.11.12 False False False 18s RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.sv-upg-vsp.qe.devcluster.openshift.com): Get "https://console-openshift-console.apps.sv-upg-vsp.qe.devcluster.openshift.com": dial tcp 172.31.248.5:443: connect: connection refused 11-01 18:30:57.915 ingress 4.11.12 True False True 5h47m The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)
Version-Release number of selected component (if applicable):
[root@13e9e59e8fe2 Development]# oc version Client Version: 4.12.0-0.nightly-2022-10-05-053337 Kustomize Version: v4.5.4 Server Version: 4.11.12 Kubernetes Version: v1.24.6+5157800 [root@13e9e59e8fe2 Development]#
How reproducible:
Load cluster by executing kube burner cluster density test with JOB_ITERATIONS=75 to create namespaces, pods and other resources and then upgrade to 4.11.12 z stream.
Steps to Reproduce:
1. Create a 20 worker node cluster using 4.10.39 release. 2. Run kube burner cluster density test with JOB_ITERATIONS=75 to create namespaces, pods and other resources (See https://github.com/cloud-bulldozer/e2e-benchmarking/blob/master/workloads/kube-burner/README.md) 3. After test is executed, upgrade cluster to 4.11.12 4. Notice cluster operators reported as degraded.
Actual results:
11-01 18:30:57.914 **************Post Action after upgrade succ**************** 11-01 18:30:57.914 11-01 18:30:57.914 Post action: #oc get node: NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 11-01 18:30:57.914 sv-upg-vsp-ptnwh-master-0 Ready master 6h2m v1.24.6+5157800 172.31.249.17 172.31.249.17 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-master-1 Ready master 6h2m v1.24.6+5157800 172.31.249.174 172.31.249.174 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-master-2 Ready master 6h2m v1.24.6+5157800 172.31.249.228 172.31.249.228 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-5c88r Ready worker 5h24m v1.24.6+5157800 172.31.249.69 172.31.249.69 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-7g4p6 Ready worker 5h24m v1.24.6+5157800 172.31.249.81 172.31.249.81 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-89bc8 Ready worker 5h23m v1.24.6+5157800 172.31.249.110 172.31.249.110 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-8gplh Ready worker 5h24m v1.24.6+5157800 172.31.249.185 172.31.249.185 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-8k6f8 Ready worker 5h49m v1.24.6+5157800 172.31.249.165 172.31.249.165 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-b4pxf Ready worker 5h24m v1.24.6+5157800 172.31.249.226 172.31.249.226 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-bn9dm Ready worker 5h24m v1.24.6+5157800 172.31.249.244 172.31.249.244 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-d7rb6 Ready worker 5h24m v1.24.6+5157800 172.31.249.37 172.31.249.37 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-f96wv Ready worker 5h49m v1.24.6+5157800 172.31.249.253 172.31.249.253 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-gj4v8 Ready worker 5h24m v1.24.6+5157800 172.31.249.137 172.31.249.137 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-gxl4w Ready worker 5h24m v1.24.6+5157800 172.31.249.187 172.31.249.187 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-kcd72 Ready worker 5h24m v1.24.6+5157800 172.31.249.254 172.31.249.254 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-prbkb Ready worker 5h24m v1.24.6+5157800 172.31.249.126 172.31.249.126 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-r8mnm Ready worker 5h23m v1.24.6+5157800 172.31.249.184 172.31.249.184 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-rgq7w Ready worker 5h24m v1.24.6+5157800 172.31.249.172 172.31.249.172 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-rnkjp Ready worker 5h24m v1.24.6+5157800 172.31.249.120 172.31.249.120 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-skjmq Ready worker 5h24m v1.24.6+5157800 172.31.249.11 172.31.249.11 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-wmsfz Ready worker 5h24m v1.24.6+5157800 172.31.249.75 172.31.249.75 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.914 sv-upg-vsp-ptnwh-worker-xkpm2 Ready worker 5h24m v1.24.6+5157800 172.31.249.216 172.31.249.216 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.915 sv-upg-vsp-ptnwh-worker-xzwj2 Ready worker 5h24m v1.24.6+5157800 172.31.249.95 172.31.249.95 Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa) 4.18.0-372.26.1.el8_6.x86_64 cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8 11-01 18:30:57.915 11-01 18:30:57.915 11-01 18:30:57.915 Post action: #oc get co:NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE 11-01 18:30:57.915 authentication 4.11.12 False False False 21s OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.sv-upg-vsp.qe.devcluster.openshift.com/healthz": dial tcp 172.31.248.5:443: connect: connection refused 11-01 18:30:57.915 baremetal 4.11.12 True False False 5h57m 11-01 18:30:57.915 cloud-controller-manager 4.11.12 True False False 6h2m 11-01 18:30:57.915 cloud-credential 4.11.12 True False False 6h2m 11-01 18:30:57.915 cluster-autoscaler 4.11.12 True False False 5h57m 11-01 18:30:57.915 config-operator 4.11.12 True False False 5h58m 11-01 18:30:57.915 console 4.11.12 False False False 18s RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.sv-upg-vsp.qe.devcluster.openshift.com): Get "https://console-openshift-console.apps.sv-upg-vsp.qe.devcluster.openshift.com": dial tcp 172.31.248.5:443: connect: connection refused 11-01 18:30:57.915 csi-snapshot-controller 4.11.12 True False False 4h46m 11-01 18:30:57.915 dns 4.11.12 True False False 5h57m 11-01 18:30:57.915 etcd 4.11.12 True False False 5h56m 11-01 18:30:57.915 image-registry 4.11.12 True False False 65m 11-01 18:30:57.915 ingress 4.11.12 True False True 5h47m The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing) 11-01 18:30:57.915 insights 4.11.12 True False False 5h51m 11-01 18:30:57.915 kube-apiserver 4.11.12 True False False 5h46m 11-01 18:30:57.915 kube-controller-manager 4.11.12 True False False 5h55m 11-01 18:30:57.915 kube-scheduler 4.11.12 True False False 5h56m 11-01 18:30:57.915 kube-storage-version-migrator 4.11.12 True False False 96m 11-01 18:30:57.915 machine-api 4.11.12 True False False 5h54m 11-01 18:30:57.915 machine-approver 4.11.12 True False False 5h58m 11-01 18:30:57.915 machine-config 4.11.12 True False False 96m 11-01 18:30:57.915 marketplace 4.11.12 True False False 5h57m 11-01 18:30:57.915 monitoring 4.11.12 True False False 5h47m 11-01 18:30:57.915 network 4.11.12 True False False 5h59m 11-01 18:30:57.915 node-tuning 4.11.12 True False False 169m 11-01 18:30:57.915 openshift-apiserver 4.11.12 True False False 5h46m 11-01 18:30:57.915 openshift-controller-manager 4.11.12 True False False 4h45m 11-01 18:30:57.915 openshift-samples 4.11.12 True False False 169m 11-01 18:30:57.915 operator-lifecycle-manager 4.11.12 True False False 5h58m 11-01 18:30:57.915 operator-lifecycle-manager-catalog 4.11.12 True False False 5h58m 11-01 18:30:57.915 operator-lifecycle-manager-packageserver 4.11.12 True False False 5h51m 11-01 18:30:57.915 service-ca 4.11.12 True False False 5h58m 11-01 18:30:57.915 storage 4.11.12 True False False 105m 11-01 18:30:57.915
Expected results:
No operators should be degraded
Additional info: