Observed upgrade ci failure in https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-multi-nightly-4.19-upgrade-from-stable-4.14-gcp-ipi-sdn-arm-f28/1897258628478406656

The upgrade failed at stage 4.18-4.19 due to one crd consoleplugins.console.openshift.io failed to update.

$ omc get clusterversion version 
NAME      VERSION                                    AVAILABLE   PROGRESSING   SINCE   STATUS version   
4.18.0-0.nightly-multi-2025-03-05-030751   True        True          4m      Unable to apply 4.19.0-0.nightly-multi-2025-03-05-061941: some cluster configuration is invalid 

{
  "lastTransitionTime": "2025-03-05T23:23:08Z",
  "message": "Could not update customresourcedefinition \"consoleplugins.console.openshift.io\" (592 of 903): the object is invalid, possibly due to local cluster configuration",
  "reason": "UpdatePayloadResourceInvalid",
  "status": "True",
  "type": "Failing"
}

2025-03-05T23:23:04.720952820Z I0305 23:23:04.720929       1 task_graph.go:564] Result of work: errs=[Could not update customresourcedefinition "consoleplugins.console.openshift.io" (592 of 903): the object is invalid, possibly due to local cluster configuration] 2025-03-05T23:23:04.720959460Z I0305 23:23:04.720948       1 sync_worker.go:1212] Update error 592 of 903: UpdatePayloadResourceInvalid Could not update customresourcedefinition "consoleplugins.console.openshift.io" (592 of 903): the object is invalid, possibly due to local cluster configuration (*errors.StatusError: CustomResourceDefinition.apiextensions.k8s.io "consoleplugins.console.openshift.io" is invalid: status.storedVersions[0]: Invalid value: "v1alpha1": must appear in spec.versions) 2025-03-05T23:23:04.721021700Z E0305 23:23:04.720992       1 sync_worker.go:690] "Unhandled Error" err="unable to synchronize image (waiting 2m46.22437854s): Could not update customresourcedefinition \"consoleplugins.console.openshift.io\" (592 of 903): the object is invalid, possibly due to local cluster configuration" logger="UnhandledError" 

    4.19.0-0.nightly-multi-2025-03-05-061941

    100%

    1. upgrade cluster from 4.14-4.15-4.16-4.17-4.18-4.19
    2.
    3.
    

    upgrade stuck at customresourcedefinition "consoleplugins.console.openshift.io"

    upgrade should succeed

must-gather logs is available https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.19-multi-nightly-4.19-upgrade-from-stable-4.14-gcp-ipi-sdn-arm-f28/1897258628478406656/artifacts/gcp-ipi-sdn-arm-f28/gather-must-gather/artifacts/must-gather.tar