-
Bug
-
Resolution: Not a Bug
-
Normal
-
None
-
4.12.z
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
No
-
None
-
None
-
Proposed
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
I had the cluster at version 4.11.37 (it was upgraded from 4.11.6 maybe up to 4.11.37 along the past 3 months)
Last week I tried to change the channel to stable-4.12 and upgrade to 4.12.26 (since it showed on the graph and on ACM it was possible)
So from ACM I changed the channel to 4.12 of the managed cluster (spoke) then initiated the upgrade to 4.12.26 it seemed to be blocked and did not go thru.
I thought it was not possible then, I wanted to upgrade to 4.11.46 which was the latest in 4.11 stable (I changed back to stable-4.11) ... since then I was not able to do much with the clusterversion object it was always complaining about this (capabilities)
oc patch clusterversion version --type merge -p '{"spec": {"channel": "stable-4.12"}}'
The ClusterVersion "version" is invalid:
* status.capabilities.enabledCapabilities[0]: Unsupported value: "CSISnapshot": supported values: "openshift-samples", "baremetal", "marketplace"
* status.capabilities.enabledCapabilities[1]: Unsupported value: "Console": supported values: "openshift-samples", "baremetal", "marketplace"
* status.capabilities.enabledCapabilities[2]: Unsupported value: "Insights": supported values: "openshift-samples", "baremetal", "marketplace"
* status.capabilities.enabledCapabilities[3]: Unsupported value: "Storage": supported values: "openshift-samples", "baremetal", "marketplace"
* status.capabilities.knownCapabilities[0]: Unsupported value: "CSISnapshot": supported values: "openshift-samples", "baremetal", "marketplace"
* status.capabilities.knownCapabilities[1]: Unsupported value: "Console": supported values: "openshift-samples", "baremetal", "marketplace"
* status.capabilities.knownCapabilities[2]: Unsupported value: "Insights": supported values: "openshift-samples", "baremetal", "marketplace"
* status.capabilities.knownCapabilities[3]: Unsupported value: "Storage": supported values: "openshift-samples", "baremetal", "marketplace"
so I reinstalled the machine with 4.11.47 : after the installation has finished ACM automatically reverted back to 4.11.37 ... I thought that would be it and ACM would be happy ... but no ... I am trying to upgrade to 4.11.47 and each time it get reverted back to 4.11.37 ,,,
here is the output from spoke cluster.
[spoke-dl3 09:51:42 must-gather]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.37 True False 16h Cluster version is 4.11.37
[spoke-dl3 09:58:06 must-gather]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.37 True False 16h Cluster version is 4.11.37
[spoke-dl3 09:58:08 must-gather]$ oc adm upgrade
Cluster version is 4.11.37
Upstream: https://api.openshift.com/api/upgrades_info/v1/graph
Channel: stable-4.11 (available channels: candidate-4.11, candidate-4.12, eus-4.12, fast-4.11, fast-4.12, stable-4.11, stable-4.12)
Recommended updates:
VERSION IMAGE
4.11.47 quay.io/openshift-release-dev/ocp-release@sha256:bbed7981242fbc034a0e75ef265040137515da3756ac2eab9d2dd46348967c06
4.11.46 quay.io/openshift-release-dev/ocp-release@sha256:88583eeaddcda4fbfdcf21f4dad86b01ff09bb010357c51f08fb24eb07fdb602
4.11.45 quay.io/openshift-release-dev/ocp-release@sha256:c6771b12bd873c0e3e5fbc7afa600d92079de6534dcb52f09cb1d22ee49608a9
4.11.44 quay.io/openshift-release-dev/ocp-release@sha256:52cbfbbeb9cc03b49c2788ac7333e63d3dae14673e01a9d8e59270f3a8390ed3
4.11.43 quay.io/openshift-release-dev/ocp-release@sha256:d11cd63a623d33bda744f1a886d4b55710db4d68878967be2ebcd5535fafa25d
4.11.42 quay.io/openshift-release-dev/ocp-release@sha256:518177a34452837920f1e77944f6afa08864537260c9f742b8c88b6157e4f901
4.11.41 quay.io/openshift-release-dev/ocp-release@sha256:400267c7f4e61c6bfa0a59571467e8bd85c9188e442cbd820cc8263809be3775
4.11.40 quay.io/openshift-release-dev/ocp-release@sha256:a6c98d2454e20cafe049972ff010262c54ea858256e6314ed7c777f266274959
4.11.39 quay.io/openshift-release-dev/ocp-release@sha256:3545730c2018e0b092d6132b31068e517cbe99b99c52c54f2a9afad61e051e3d
4.11.38 quay.io/openshift-release-dev/ocp-release@sha256:4f6089b62f54887ecc14684fafbe235a12f9d0c6b77379dee6ccc3675de6ed6d
[spoke-dl3 10:51:59 must-gather]$ oc adm upgrade --to-latest
Requesting update to 4.11.47
[spoke-dl3 10:52:40 must-gather]$ oc get clusterversion -w
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.37 True False 17h Cluster version is 4.11.37
version 4.11.37 True False 17h Cluster version is 4.11.37
version 4.11.37 True True 0s Working towards 4.11.47: 3 of 806 done (0% complete)
version 4.11.37 True True 0s Working towards 4.11.47: 5 of 806 done (0% complete)
version 4.11.37 True True 0s Working towards 4.11.47: 6 of 806 done (0% complete)
version 4.11.37 True True 0s Working towards 4.11.47: 9 of 806 done (1% complete)
version 4.11.37 True True 6s Working towards 4.11.47
version 4.11.37 True True 6s Working towards 4.11.47: 2 of 806 done (0% complete)
version 4.11.37 True True 6s Working towards 4.11.47: 4 of 806 done (0% complete)
version 4.11.37 True True 6s Working towards 4.11.47: 6 of 806 done (0% complete)
version 4.11.37 True True 21s Working towards 4.11.47: 104 of 806 done (12% complete)
[spoke-dl3 10:54:54 must-gather]$ oc get clusterversion -w
The connection to the server api.dl3.hpecloud.org:6443 was refused - did you specify the right host or port?
[spoke-dl3 10:55:15 must-gather]$ oc get clusterversion -w
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.37 True True 3m43s Working towards 4.11.47: 105 of 806 done (13% complete)
version 4.11.37 True True 3m48s Working towards 4.11.47: 105 of 806 done (13% complete)
version 4.11.37 True True 4m1s Working towards 4.11.47: 105 of 806 done (13% complete)
version 4.11.37 True True 4m15s Working towards 4.11.47: 105 of 806 done (13% complete)
version 4.11.37 True True 4m15s Working towards 4.11.37: 2 of 806 done (0% complete)
version 4.11.37 True True 4m15s Working towards 4.11.37: 4 of 806 done (0% complete)
version 4.11.37 True True 4m15s Working towards 4.11.37: 6 of 806 done (0% complete)
version 4.11.37 True True 4m15s Working towards 4.11.37: 9 of 806 done (1% complete)
[spoke-dl3 10:57:21 must-gather]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.37 True True 8m53s Working towards 4.11.37: 104 of 806 done (12% complete)
[spoke-dl3 11:01:46 must-gather]$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.37 True False 5m24s Cluster version is 4.11.37
[spoke-dl3 11:09:34 must-gather]$ ls -lrth
here is in console GUI of spoke cluster what it says:
Update history
There is a threshold for rendering update data which may cause gaps in the information below.
Version State Started Completed Release notes
4.11.37 Completed Aug 25, 2023, 10:57 AM Aug 25, 2023, 11:04 AM View release notes
4.11.47 Partial Aug 25, 2023, 10:52 AM Aug 25, 2023, 10:57 AM View release notes
4.11.37 Completed Aug 24, 2023, 5:42 PM Aug 24, 2023, 5:43 PM View release notes
4.11.47 Partial Aug 24, 2023, 5:42 PM Aug 24, 2023, 5:42 PM View release notes
4.11.37 Completed Aug 24, 2023, 4:37 PM Aug 24, 2023, 5:25 PM View release notes
4.11.47 Completed Aug 24, 2023, 3:38 PM Aug 24, 2023, 4:30 PM View release notes
I will attach must gather from both clusters
Version-Release number of selected component (if applicable):
4.12
How reproducible:
An ACM cluster : ACM 2.6 (OCP 4.11.4x) or 2.7 (OCP 4.12.x): A spoke cluster with OCP : 4.11.37 installed with ZTP: from ACM change the channel to stable-4.12 and launch the upgrade from ACM GUI too. and then a little while (not sure maybe 2 to 3 minutes or less) I suspected that upgrade did not go thru : I reverted to stable-4.11 and tried to upgrade to 4.11.45 (anything newer than 4.11.37) you will see that any upgrade will fail and revert back to 4.11.37 even with deleting and reinstalling the cluster
Steps to Reproduce:
1. I even have removed the cluster from siteconfig and cleaned up all resources in namespace of the spoken (namespace was deleted) 2. then reinstalled 4.11.47 : after reinstall has completed : ACM reverted it back to 4.11.37 automatically and not able to upgrade the cluster at all. 3.
Actual results:
no upgrade is possible ... always reverted back to 4.11.37 either be it from ACM GUI or from spoke oc commands.
Expected results:
upgrade should proceed on spoke cluster
Additional info:
for more details open muster-gather in case : https://access.redhat.com/support/cases/#/case/03597448