Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-48493

[release-4.17] TP cluster micro-upgrade fails, waiting on cluster-api

    • Important
    • No
    • CLOUD Sprint 265
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • N/A
    • Release Note Not Required
    • Done

      Description of problem:

      On azure(or vsphere) TP cluster upgrade failed from 4.15.0-rc.5-> 4.15.0-rc.7 or 4.15.0-rc.4-> 4.15.0-rc.5, stuck in cluster-api.
      Seems this only happened on platforms don't support capi, this couldn't be reproduced on aws and gcp, .

      Version-Release number of selected component (if applicable):

          4.15.0-rc.5-> 4.15.0-rc.7 or 4.15.0-rc.4-> 4.15.0-rc.5

      How reproducible:

          always

      Steps to Reproduce:

          1.Build a TP cluster 4.15.0-rc.5 on azure(or vsphere)
          2.Upgrade to 4.15.0-rc.7     
          3.
          

      Actual results:

      Upgrade stuck in cluster-api. 
      must-gather: https://drive.google.com/file/d/12ykhEVZvqY_0eNdLwJOWFSxTSdQQrm_y/view?usp=sharing
      
      $ oc get clusterversion       NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS version   4.15.0-rc.5   True        True          82m     Working towards 4.15.0-rc.7: 257 of 929 done (27% complete), waiting on cluster-api
      
      I0222 04:53:18.733907       1 sync_worker.go:1134] Update error 198 of 929: ClusterOperatorUpdating Cluster operator cluster-api is updating versions (*errors.errorString: cluster operator cluster-api is available and not degraded but has not finished updating to target version) E0222 04:53:18.733944       1 sync_worker.go:638] unable to synchronize image (waiting 2m44.892272217s): Cluster operator cluster-api is updating versions
      
      $ oc get co     
      NAME                                       VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      authentication                             4.15.0-rc.5   True        False         False      99m
      baremetal                                  4.15.0-rc.5   True        False         False      123m
      cloud-controller-manager                   4.15.0-rc.7   True        False         False      128m
      cloud-credential                           4.15.0-rc.5   True        False         False      135m
      cluster-api                                4.15.0-rc.5   True        False         False      124m
      cluster-autoscaler                         4.15.0-rc.5   True        False         False      123m
      config-operator                            4.15.0-rc.7   True        False         False      124m
      console                                    4.15.0-rc.5   True        False         False      101m
      control-plane-machine-set                  4.15.0-rc.7   True        False         False      113m
      csi-snapshot-controller                    4.15.0-rc.5   True        False         False      112m
      dns                                        4.15.0-rc.5   True        False         False      115m
      etcd                                       4.15.0-rc.7   True        False         False      122m
      image-registry                             4.15.0-rc.5   True        False         False      107m
      ingress                                    4.15.0-rc.5   True        False         False      106m
      insights                                   4.15.0-rc.5   True        False         False      118m
      kube-apiserver                             4.15.0-rc.7   True        False         False      108m
      kube-controller-manager                    4.15.0-rc.7   True        False         False      121m
      kube-scheduler                             4.15.0-rc.7   True        False         False      120m
      kube-storage-version-migrator              4.15.0-rc.5   True        False         False      115m
      machine-api                                4.15.0-rc.7   True        False         False      111m
      machine-approver                           4.15.0-rc.5   True        False         False      124m
      machine-config                             4.15.0-rc.5   True        False         False      121m
      marketplace                                4.15.0-rc.5   True        False         False      123m
      monitoring                                 4.15.0-rc.5   True        False         False      106m
      network                                    4.15.0-rc.5   True        False         False      126m
      node-tuning                                4.15.0-rc.5   True        False         False      112m
      olm                                        4.15.0-rc.5   True        False         False      106m
      openshift-apiserver                        4.15.0-rc.5   True        False         False      115m
      openshift-controller-manager               4.15.0-rc.5   True        False         False      115m
      openshift-samples                          4.15.0-rc.5   True        False         False      111m
      operator-lifecycle-manager                 4.15.0-rc.5   True        False         False      123m
      operator-lifecycle-manager-catalog         4.15.0-rc.5   True        False         False      123m
      operator-lifecycle-manager-packageserver   4.15.0-rc.5   True        False         False      112m
      platform-operators-aggregated              4.15.0-rc.5   True        False         False      73m
      service-ca                                 4.15.0-rc.5   True        False         False      124m
      storage                                    4.15.0-rc.5   True        False         False      107m  

      Expected results:

      Upgrade is successful

      Additional info:

       upgrade succeed from 4.15.0-rc.3-> 4.15.0-rc.4

            [OCPBUGS-48493] [release-4.17] TP cluster micro-upgrade fails, waiting on cluster-api

            Damiano Donati created issue -
            Damiano Donati made changes -
            Link New: This issue clones OCPBUGS-48429 [ OCPBUGS-48429 ]
            Damiano Donati made changes -
            Target Version Original: 4.18.0 [ 12431397 ] New: 4.17.z [ 12428296 ]
            OpenShift Jira Bot made changes -
            Assignee Original: Damiano Donati [ ddonati@redhat.com ] New: Joel Speed [ joelspeed ]
            Damiano Donati made changes -
            Link New: This issue depends on OCPBUGS-48429 [ OCPBUGS-48429 ]
            Damiano Donati made changes -
            Assignee Original: Joel Speed [ joelspeed ] New: Damiano Donati [ ddonati@redhat.com ]
            Damiano Donati made changes -
            Status Original: New [ 10016 ] New: ASSIGNED [ 14452 ]
            Damiano Donati made changes -
            Sprint New: CLOUD Sprint 265 [ 67124 ]
            OpenShift Prow Bot made changes -
            Remote Link New: This issue links to "openshift/cluster-capi-operator#250: [release-4.17] OCPBUGS-48493: fix: always update clusteroperator status versions when differing (Web Link)" [ 1905899 ]
            OpenShift Prow Bot made changes -
            Status Original: ASSIGNED [ 14452 ] New: POST [ 15726 ]
            OpenShift Prow Bot made changes -
            Status Original: POST [ 15726 ] New: MODIFIED [ 14454 ]
            ART Bot made changes -
            Status Original: MODIFIED [ 14454 ] New: ON_QA [ 15723 ]
            Damiano Donati made changes -
            Summary Original: [release-4.17] TP cluster upgrade from 4.15.0-rc.5-> 4.15.0-rc.7 failed, waiting on cluster-api New: [release-4.17] TP cluster micro-upgrade fails, waiting on cluster-api
            Zhaohua Sun made changes -
            Status Original: ON_QA [ 15723 ] New: Verified [ 10015 ]
            Errata Tool made changes -
            Remote Link New: This issue links to "RHBA-2025:0654 (Web Link)" [ 1915057 ]
            Tami Love made changes -
            Release Note Status Original: In Progress [ 30960 ] New: Done [ 30963 ]
            Release Note Text New: N/A
            W. Trevor King made changes -
            Link New: This issue is duplicated by OCPBUGS-49321 [ OCPBUGS-49321 ]
            Errata Tool made changes -
            Resolution New: Done-Errata [ 10803 ]
            Status Original: Verified [ 10015 ] New: Closed [ 6 ]
            Damiano Donati made changes -
            Link New: This issue is cloned by OCPBUGS-49603 [ OCPBUGS-49603 ]
            Damiano Donati made changes -
            Link New: This issue is depended on by OCPBUGS-49603 [ OCPBUGS-49603 ]

              ddonati@redhat.com Damiano Donati
              rhn-support-zhsun Zhaohua Sun
              Zhaohua Sun Zhaohua Sun
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: