Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-3044

Vsphere: console and ingress operators are reporting degraded during loaded upgrade from 4.10.39 to 4.11.12

XMLWordPrintable

    • None
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      While running loaded upgrade from 4.10.z to 4.11.z release on a 20 worker node vSphere cluster, noticed that the cluster operators are reported as degraded after upgrading cluster from 4.10.39 to 4.11.12.
      
      #oc get co:NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      
      11-01 18:30:57.915  console                                    4.11.12   False       False         False      18s     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.sv-upg-vsp.qe.devcluster.openshift.com): Get "https://console-openshift-console.apps.sv-upg-vsp.qe.devcluster.openshift.com": dial tcp 172.31.248.5:443: connect: connection refused
      
      11-01 18:30:57.915  ingress                                    4.11.12   True        False         True       5h47m   The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)
      
      

      Version-Release number of selected component (if applicable):

      [root@13e9e59e8fe2 Development]# oc version
      Client Version: 4.12.0-0.nightly-2022-10-05-053337
      Kustomize Version: v4.5.4
      Server Version: 4.11.12
      Kubernetes Version: v1.24.6+5157800
      [root@13e9e59e8fe2 Development]# 

      How reproducible:

      Load cluster by executing kube burner cluster density test with JOB_ITERATIONS=75 to create namespaces, pods and other resources and then upgrade to 4.11.12 z stream.
      
      

      Steps to Reproduce:

      1. Create a 20 worker node cluster using 4.10.39 release. 
      2. Run kube burner cluster density test with JOB_ITERATIONS=75 to create namespaces, pods and other resources (See https://github.com/cloud-bulldozer/e2e-benchmarking/blob/master/workloads/kube-burner/README.md)
      3. After test is executed, upgrade cluster to 4.11.12
      4. Notice cluster operators reported as degraded.

      Actual results:

      11-01 18:30:57.914  **************Post Action after upgrade succ****************
      11-01 18:30:57.914  
      11-01 18:30:57.914  Post action: #oc get node: NAME                            STATUS   ROLES    AGE     VERSION           INTERNAL-IP      EXTERNAL-IP      OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-master-0       Ready    master   6h2m    v1.24.6+5157800   172.31.249.17    172.31.249.17    Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-master-1       Ready    master   6h2m    v1.24.6+5157800   172.31.249.174   172.31.249.174   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-master-2       Ready    master   6h2m    v1.24.6+5157800   172.31.249.228   172.31.249.228   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-5c88r   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.69    172.31.249.69    Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-7g4p6   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.81    172.31.249.81    Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-89bc8   Ready    worker   5h23m   v1.24.6+5157800   172.31.249.110   172.31.249.110   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-8gplh   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.185   172.31.249.185   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-8k6f8   Ready    worker   5h49m   v1.24.6+5157800   172.31.249.165   172.31.249.165   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-b4pxf   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.226   172.31.249.226   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-bn9dm   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.244   172.31.249.244   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-d7rb6   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.37    172.31.249.37    Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-f96wv   Ready    worker   5h49m   v1.24.6+5157800   172.31.249.253   172.31.249.253   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-gj4v8   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.137   172.31.249.137   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-gxl4w   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.187   172.31.249.187   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-kcd72   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.254   172.31.249.254   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-prbkb   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.126   172.31.249.126   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-r8mnm   Ready    worker   5h23m   v1.24.6+5157800   172.31.249.184   172.31.249.184   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-rgq7w   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.172   172.31.249.172   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-rnkjp   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.120   172.31.249.120   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-skjmq   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.11    172.31.249.11    Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-wmsfz   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.75    172.31.249.75    Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.914  sv-upg-vsp-ptnwh-worker-xkpm2   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.216   172.31.249.216   Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.915  sv-upg-vsp-ptnwh-worker-xzwj2   Ready    worker   5h24m   v1.24.6+5157800   172.31.249.95    172.31.249.95    Red Hat Enterprise Linux CoreOS 411.86.202210201510-0 (Ootpa)   4.18.0-372.26.1.el8_6.x86_64   cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
      11-01 18:30:57.915  
      11-01 18:30:57.915  
      11-01 18:30:57.915  Post action: #oc get co:NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      11-01 18:30:57.915  authentication                             4.11.12   False       False         False      21s     OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.sv-upg-vsp.qe.devcluster.openshift.com/healthz": dial tcp 172.31.248.5:443: connect: connection refused
      11-01 18:30:57.915  baremetal                                  4.11.12   True        False         False      5h57m   
      11-01 18:30:57.915  cloud-controller-manager                   4.11.12   True        False         False      6h2m    
      11-01 18:30:57.915  cloud-credential                           4.11.12   True        False         False      6h2m    
      11-01 18:30:57.915  cluster-autoscaler                         4.11.12   True        False         False      5h57m   
      11-01 18:30:57.915  config-operator                            4.11.12   True        False         False      5h58m   
      11-01 18:30:57.915  console                                    4.11.12   False       False         False      18s     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.sv-upg-vsp.qe.devcluster.openshift.com): Get "https://console-openshift-console.apps.sv-upg-vsp.qe.devcluster.openshift.com": dial tcp 172.31.248.5:443: connect: connection refused
      11-01 18:30:57.915  csi-snapshot-controller                    4.11.12   True        False         False      4h46m   
      11-01 18:30:57.915  dns                                        4.11.12   True        False         False      5h57m   
      11-01 18:30:57.915  etcd                                       4.11.12   True        False         False      5h56m   
      11-01 18:30:57.915  image-registry                             4.11.12   True        False         False      65m     
      11-01 18:30:57.915  ingress                                    4.11.12   True        False         True       5h47m   The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)
      11-01 18:30:57.915  insights                                   4.11.12   True        False         False      5h51m   
      11-01 18:30:57.915  kube-apiserver                             4.11.12   True        False         False      5h46m   
      11-01 18:30:57.915  kube-controller-manager                    4.11.12   True        False         False      5h55m   
      11-01 18:30:57.915  kube-scheduler                             4.11.12   True        False         False      5h56m   
      11-01 18:30:57.915  kube-storage-version-migrator              4.11.12   True        False         False      96m     
      11-01 18:30:57.915  machine-api                                4.11.12   True        False         False      5h54m   
      11-01 18:30:57.915  machine-approver                           4.11.12   True        False         False      5h58m   
      11-01 18:30:57.915  machine-config                             4.11.12   True        False         False      96m     
      11-01 18:30:57.915  marketplace                                4.11.12   True        False         False      5h57m   
      11-01 18:30:57.915  monitoring                                 4.11.12   True        False         False      5h47m   
      11-01 18:30:57.915  network                                    4.11.12   True        False         False      5h59m   
      11-01 18:30:57.915  node-tuning                                4.11.12   True        False         False      169m    
      11-01 18:30:57.915  openshift-apiserver                        4.11.12   True        False         False      5h46m   
      11-01 18:30:57.915  openshift-controller-manager               4.11.12   True        False         False      4h45m   
      11-01 18:30:57.915  openshift-samples                          4.11.12   True        False         False      169m    
      11-01 18:30:57.915  operator-lifecycle-manager                 4.11.12   True        False         False      5h58m   
      11-01 18:30:57.915  operator-lifecycle-manager-catalog         4.11.12   True        False         False      5h58m   
      11-01 18:30:57.915  operator-lifecycle-manager-packageserver   4.11.12   True        False         False      5h51m   
      11-01 18:30:57.915  service-ca                                 4.11.12   True        False         False      5h58m   
      11-01 18:30:57.915  storage                                    4.11.12   True        False         False      105m    
      11-01 18:30:57.915  

      Expected results:

      No operators should be degraded

      Additional info:

       

              rhn-support-misalunk Miheer Salunke
              svetsa@redhat.com Sharada Vetsa
              Hongan Li Hongan Li
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: