-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
4.20
-
Quality / Stability / Reliability
-
False
-
-
2
-
Critical
-
None
-
None
-
None
-
Approved
-
MCO Sprint 275
-
1
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Observed upgrade ci failures in 4.15-4.20 upgrade(refer to [1]) and 4.14-4.20 upgrade(refer to [2]).The upgrade failed at stage 4.19-4.20 due to one crd machineconfignodes.machineconfiguration.openshift.io failed to update. The previous upgrades from 4.15-4.19/4.14-4.19 succeed.
$ omc get clusterversion version -ojson|jq -r '.status.history[]|.state + " " + .version'
Partial 4.20.0-ec.4
Completed 4.19.4
Completed 4.18.20
Completed 4.17.36
Completed 4.16.44
Completed 4.15.54
{
"lastTransitionTime": "2025-07-18T10:09:54Z",
"message": "Could not update customresourcedefinition \"machineconfignodes.machineconfiguration.openshift.io\" (789 of 936): the object is invalid, possibly due to local cluster configuration",
"reason": "UpdatePayloadResourceInvalid",
"status": "True",
"type": "Failing"
},
$ omc -n openshift-cluster-version logs cluster-version-operator-549f995b87-p6v79|grep customresourcedefinition| tail -n1
2025-07-18T10:04:08.379031100Z E0718 10:04:08.378875 1 task.go:128] "Unhandled Error" err="error running apply for customresourcedefinition \"machineconfignodes.machineconfiguration.openshift.io\" (789 of 936): CustomResourceDefinition.apiextensions.k8s.io \"machineconfignodes.machineconfiguration.openshift.io\" is invalid: status.storedVersions[0]: Invalid value: \"v1alpha1\": must appear in spec.versions" logger="UnhandledError"
[1] https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.20-multi-stable-4.20-upgrade-from-stable-4.15-azure-ipi-ovn-ipsec-arm-mixarch-f28/1946004383644782592
[2] https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.20-multi-nightly-4.20-upgrade-from-stable-4.14-azure-ipi-basecap-vset-additionalcap-arm-f28/1945725047163326464
Version-Release number of selected component (if applicable):
4.20.0-0.nightly-multi-2025-07-17-001658 4.20.0-ec.4
How reproducible:
100%
Steps to Reproduce:
1. chain upgrade cluster from 4.14 to 4.20 or 4.15 to 4.20
2.
3.
Actual results:
The final hop 4.19-4.20 upgrade stuck at customresourcedefinition "machineconfignodes.machineconfiguration.openshift.io"
Expected results:
upgrade should succeed
Additional info:
must-gather logs are available in https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.20-multi-stable-4.20-upgrade-from-stable-4.15-azure-ipi-ovn-ipsec-arm-mixarch-f28/1946004383644782592/artifacts/azure-ipi-ovn-ipsec-arm-mixarch-f28/gather-must-gather/artifacts/must-gather.tar
https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.20-multi-nightly-4.20-upgrade-from-stable-4.14-azure-ipi-basecap-vset-additionalcap-arm-f28/1945725047163326464/artifacts/azure-ipi-basecap-vset-additionalcap-arm-f28/gather-must-gather/artifacts/must-gather.tar
- blocks
-
MCO-1720 Backport MCN feature gate to default in 4.19
-
- Closed
-
- incorporates
-
-
- Verified
-
-
-
- Closed
-
- relates to
-
-
- Closed
-
