Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18140

ACM refusing to upgrade spoke cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Normal Normal
    • None
    • 4.12.z
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • No
    • None
    • None
    • Proposed
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      I had the cluster at version 4.11.37 (it was upgraded from 4.11.6 maybe up to 4.11.37 along the past 3 months) 
      Last week I tried to change the channel to stable-4.12 and upgrade to 4.12.26 (since it showed on the graph and on ACM it was possible)
      So from ACM I changed the channel to 4.12 of the managed cluster (spoke) then initiated the upgrade to 4.12.26 it seemed to be blocked and did not go thru.
      I thought it was not possible then, I wanted to upgrade to 4.11.46 which was the latest in 4.11 stable (I changed back to stable-4.11) ... since then I was not able to do much with the clusterversion object it was always complaining about this (capabilities) 
      
      oc patch clusterversion version --type merge -p '{"spec": {"channel": "stable-4.12"}}'
      The ClusterVersion "version" is invalid:
      * status.capabilities.enabledCapabilities[0]: Unsupported value: "CSISnapshot": supported values: "openshift-samples", "baremetal", "marketplace"
      * status.capabilities.enabledCapabilities[1]: Unsupported value: "Console": supported values: "openshift-samples", "baremetal", "marketplace"
      * status.capabilities.enabledCapabilities[2]: Unsupported value: "Insights": supported values: "openshift-samples", "baremetal", "marketplace"
      * status.capabilities.enabledCapabilities[3]: Unsupported value: "Storage": supported values: "openshift-samples", "baremetal", "marketplace"
      * status.capabilities.knownCapabilities[0]: Unsupported value: "CSISnapshot": supported values: "openshift-samples", "baremetal", "marketplace"
      * status.capabilities.knownCapabilities[1]: Unsupported value: "Console": supported values: "openshift-samples", "baremetal", "marketplace"
      * status.capabilities.knownCapabilities[2]: Unsupported value: "Insights": supported values: "openshift-samples", "baremetal", "marketplace"
      * status.capabilities.knownCapabilities[3]: Unsupported value: "Storage": supported values: "openshift-samples", "baremetal", "marketplace"
      
      so I reinstalled the machine with 4.11.47 : after the installation has finished ACM automatically reverted back to 4.11.37 ... I thought that would be it and ACM would be happy ... but no ... I am trying to upgrade to 4.11.47 and each time it get reverted back to 4.11.37 ,,, 
      
      here is the output from spoke cluster.
      [spoke-dl3 09:51:42 must-gather]$ oc get clusterversion
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.11.37   True        False         16h     Cluster version is 4.11.37
      [spoke-dl3 09:58:06 must-gather]$ oc get clusterversion
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.11.37   True        False         16h     Cluster version is 4.11.37
      [spoke-dl3 09:58:08 must-gather]$ oc adm upgrade 
      Cluster version is 4.11.37
      
      Upstream: https://api.openshift.com/api/upgrades_info/v1/graph
      Channel: stable-4.11 (available channels: candidate-4.11, candidate-4.12, eus-4.12, fast-4.11, fast-4.12, stable-4.11, stable-4.12)
      
      Recommended updates:
      
        VERSION     IMAGE
        4.11.47     quay.io/openshift-release-dev/ocp-release@sha256:bbed7981242fbc034a0e75ef265040137515da3756ac2eab9d2dd46348967c06
        4.11.46     quay.io/openshift-release-dev/ocp-release@sha256:88583eeaddcda4fbfdcf21f4dad86b01ff09bb010357c51f08fb24eb07fdb602
        4.11.45     quay.io/openshift-release-dev/ocp-release@sha256:c6771b12bd873c0e3e5fbc7afa600d92079de6534dcb52f09cb1d22ee49608a9
        4.11.44     quay.io/openshift-release-dev/ocp-release@sha256:52cbfbbeb9cc03b49c2788ac7333e63d3dae14673e01a9d8e59270f3a8390ed3
        4.11.43     quay.io/openshift-release-dev/ocp-release@sha256:d11cd63a623d33bda744f1a886d4b55710db4d68878967be2ebcd5535fafa25d
        4.11.42     quay.io/openshift-release-dev/ocp-release@sha256:518177a34452837920f1e77944f6afa08864537260c9f742b8c88b6157e4f901
        4.11.41     quay.io/openshift-release-dev/ocp-release@sha256:400267c7f4e61c6bfa0a59571467e8bd85c9188e442cbd820cc8263809be3775
        4.11.40     quay.io/openshift-release-dev/ocp-release@sha256:a6c98d2454e20cafe049972ff010262c54ea858256e6314ed7c777f266274959
        4.11.39     quay.io/openshift-release-dev/ocp-release@sha256:3545730c2018e0b092d6132b31068e517cbe99b99c52c54f2a9afad61e051e3d
        4.11.38     quay.io/openshift-release-dev/ocp-release@sha256:4f6089b62f54887ecc14684fafbe235a12f9d0c6b77379dee6ccc3675de6ed6d
      [spoke-dl3 10:51:59 must-gather]$ oc adm upgrade --to-latest 
      Requesting update to 4.11.47
      [spoke-dl3 10:52:40 must-gather]$ oc get clusterversion -w
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.11.37   True        False         17h     Cluster version is 4.11.37
      version   4.11.37   True        False         17h     Cluster version is 4.11.37
      version   4.11.37   True        True          0s      Working towards 4.11.47: 3 of 806 done (0% complete)
      version   4.11.37   True        True          0s      Working towards 4.11.47: 5 of 806 done (0% complete)
      version   4.11.37   True        True          0s      Working towards 4.11.47: 6 of 806 done (0% complete)
      version   4.11.37   True        True          0s      Working towards 4.11.47: 9 of 806 done (1% complete)
      version   4.11.37   True        True          6s      Working towards 4.11.47
      version   4.11.37   True        True          6s      Working towards 4.11.47: 2 of 806 done (0% complete)
      version   4.11.37   True        True          6s      Working towards 4.11.47: 4 of 806 done (0% complete)
      version   4.11.37   True        True          6s      Working towards 4.11.47: 6 of 806 done (0% complete)
      version   4.11.37   True        True          21s     Working towards 4.11.47: 104 of 806 done (12% complete)
      [spoke-dl3 10:54:54 must-gather]$ oc get clusterversion -w
      The connection to the server api.dl3.hpecloud.org:6443 was refused - did you specify the right host or port?
      [spoke-dl3 10:55:15 must-gather]$ oc get clusterversion -w
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.11.37   True        True          3m43s   Working towards 4.11.47: 105 of 806 done (13% complete)
      version   4.11.37   True        True          3m48s   Working towards 4.11.47: 105 of 806 done (13% complete)
      version   4.11.37   True        True          4m1s    Working towards 4.11.47: 105 of 806 done (13% complete)
      version   4.11.37   True        True          4m15s   Working towards 4.11.47: 105 of 806 done (13% complete)
      version   4.11.37   True        True          4m15s   Working towards 4.11.37: 2 of 806 done (0% complete)
      version   4.11.37   True        True          4m15s   Working towards 4.11.37: 4 of 806 done (0% complete)
      version   4.11.37   True        True          4m15s   Working towards 4.11.37: 6 of 806 done (0% complete)
      version   4.11.37   True        True          4m15s   Working towards 4.11.37: 9 of 806 done (1% complete)
      [spoke-dl3 10:57:21 must-gather]$ oc get clusterversion 
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.11.37   True        True          8m53s   Working towards 4.11.37: 104 of 806 done (12% complete)
      [spoke-dl3 11:01:46 must-gather]$ oc get clusterversion 
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.11.37   True        False         5m24s   Cluster version is 4.11.37
      [spoke-dl3 11:09:34 must-gather]$ ls -lrth
      
      here is in console GUI of spoke cluster what it says: 
      
      Update history
      There is a threshold for rendering update data which may cause gaps in the information below.
      
      Version	State			Started						Completed					Release notes
      4.11.37	Completed		Aug 25, 2023, 10:57 AM		Aug 25, 2023, 11:04 AM		View release notes
      4.11.47	Partial			Aug 25, 2023, 10:52 AM		Aug 25, 2023, 10:57 AM		View release notes
      4.11.37	Completed		Aug 24, 2023, 5:42 PM		Aug 24, 2023, 5:43 PM		View release notes
      4.11.47	Partial			Aug 24, 2023, 5:42 PM		Aug 24, 2023, 5:42 PM		View release notes
      4.11.37	Completed		Aug 24, 2023, 4:37 PM		Aug 24, 2023, 5:25 PM		View release notes
      4.11.47	Completed		Aug 24, 2023, 3:38 PM		Aug 24, 2023, 4:30 PM		View release notes
      
      I will attach must gather from both clusters

      Version-Release number of selected component (if applicable):

      4.12

      How reproducible:

      An ACM cluster : ACM 2.6 (OCP 4.11.4x) or 2.7 (OCP 4.12.x):
      A spoke cluster with OCP : 4.11.37 installed with ZTP: from ACM change the channel to stable-4.12 and launch the upgrade from ACM GUI too. and then a little while (not sure maybe 2 to 3 minutes or less) I suspected that upgrade did not go thru : I reverted to stable-4.11 and tried to upgrade to 4.11.45 (anything newer than 4.11.37)
      you will see that any upgrade will fail and revert back to 4.11.37 even with deleting and reinstalling the cluster

      Steps to Reproduce:

      1. I even have removed the cluster from siteconfig and cleaned up all resources in namespace of the spoken (namespace was deleted)
      2. then reinstalled 4.11.47 : after reinstall has completed : ACM reverted it back to 4.11.37 automatically and not able to upgrade the cluster at all.
      3.
      

      Actual results:

      no upgrade is possible ... always reverted back to 4.11.37 either be it from ACM GUI or from spoke oc commands.

      Expected results:

      upgrade should proceed on spoke cluster

      Additional info:

      for more details open muster-gather in case : https://access.redhat.com/support/cases/#/case/03597448

              lmohanty@redhat.com Lalatendu Mohanty
              lhalleb@redhat.com Lazhar Halleb
              None
              None
              Jia Liu Jia Liu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: