Create 4.18 image: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/release-openshift-origin-installer-launch-aws-modern/1883694520944562176 oc image info registry.build05.ci.openshift.org/ci-ln-4d1ld22/release:latest | grep 'Digest:' | tr -d ' ' | cut -d ':' -f2- [jianl@jianl-thinkpadt14gen4 417]$ oc image info registry.build05.ci.openshift.org/ci-ln-4d1ld22/release:latest | grep 'Digest:' | tr -d ' ' | cut -d ':' -f2- sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f [jianl@jianl-thinkpadt14gen4 417]$ launch 4.17,openshift/cluster-version-operator#1139 aws https://prow.ci.openshift.org/view/gs/test-platform-results/logs/release-openshift-origin-installer-launch-aws-modern/1883737855126147072 [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest True False 16m Cluster version is 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f error: cannot refresh available updates: Reason: NoChannel Message: The update channel has not been configured. specify --allow-explicit-upgrade to continue with the update. *********************Upgrade to 4.18************************* [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway Requested update to release image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f --allow-explicit-upgrade --force warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Requested update to release image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest True True 8s Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 6 of 902 done (0% complete) [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade info: An upgrade is in progress. Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 6 of 902 done (0% complete) Upgradeable=False Reason: UpdateInProgress Message: An update is already in progress and the details are in the Progressing condition warning: Cannot display available updates: Reason: NoChannel Message: The update channel has not been configured. [jianl@jianl-thinkpadt14gen4 417]$ ******************Upgrade to 4.19.0-ec.1****************** [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 error: cannot refresh available updates: Reason: NoChannel Message: The update channel has not been configured. specify --allow-explicit-upgrade to continue with the update. [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway error: the cluster is already upgrading: Reason: ClusterOperatorsUpdating Message: Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 111 of 902 done (12% complete), waiting on etcd, kube-apiserver If you want to upgrade anyway, use --allow-upgrade-with-warnings. [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 --allow-explicit-upgrade --allow-upgrade-with-warnings warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway warning: --allow-upgrade-with-warnings is bypassing: the cluster is already upgrading: Reason: ClusterOperatorsUpdating Message: Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 111 of 902 done (12% complete), waiting on etcd, kube-apiserver Requested update to release image quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest True True 3m32s Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 111 of 902 done (12% complete), waiting on etcd, kube-apiserver [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade info: An upgrade is in progress. Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 111 of 902 done (12% complete), waiting on etcd, kube-apiserver Upgradeable=False Reason: MultipleReasons Message: Cluster should not be upgraded between minor versions for multiple reasons: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade,UpdateInProgress * Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade. * An update is already in progress and the details are in the Progressing condition ReleaseAccepted=False Reason: PreconditionChecks Message: Preconditions failed for payload loaded version="4.19.0-ec.1" image="quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9": Multiple precondition checks failed: * Precondition "ClusterVersionUpgradeable" failed because of "MultipleReasons": Cluster should not be upgraded between minor versions for multiple reasons: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade,UpdateInProgress * Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade. * An update is already in progress and the details are in the Progressing condition * Precondition "ClusterVersionRecommendedUpdate" failed because of "NoChannel": Configured channel is unset, so the recommended status of updating from 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest to 4.19.0-ec.1 is unknown. warning: Cannot display available updates: Reason: NoChannel Message: The update channel has not been configured. [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade status Unable to fetch alerts, ignoring alerts in 'Update Health': failed to get alerts from Thanos: no token is currently in use for this session = Control Plane = Assessment: Progressing Target Version: 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest (from 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest) Updating: etcd, kube-apiserver Completion: 3% (1 operators updated, 2 updating, 30 waiting) Duration: 4m35s (Est. Time Remaining: 1h6m) Operator Health: 33 Healthy Control Plane Nodes NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-14-50.ec2.internal Outdated Pending 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest ? ip-10-0-20-3.ec2.internal Outdated Pending 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest ? ip-10-0-91-67.ec2.internal Outdated Pending 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest ? = Worker Upgrade = WORKER POOL ASSESSMENT COMPLETION STATUS worker Pending 0% (0/3) 3 Available, 0 Progressing, 0 Draining Worker Pool Nodes: worker NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-12-178.ec2.internal Outdated Pending 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest ? ip-10-0-45-171.ec2.internal Outdated Pending 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest ? ip-10-0-87-198.ec2.internal Outdated Pending 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest ? = Update Health = SINCE LEVEL IMPACT MESSAGE 4m35s Info None Update is proceeding well [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc get pod -n openshift-console-operator NAME READY STATUS RESTARTS AGE console-operator-76464cc8bc-7mxkz 1/1 Running 3 (34m ago) 36m [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc logs pod/console-operator-76464cc8bc-7mxkz -n openshift-console-operator > cvo_2.log [jianl@jianl-thinkpadt14gen4 417]$ ****************** Console UI Is Broken ****************** [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion version -oyaml apiVersion: config.openshift.io/v1 kind: ClusterVersion metadata: creationTimestamp: "2025-01-27T05:09:18Z" generation: 5 name: version resourceVersion: "36626" uid: 38a7a0a7-3226-4659-844d-3e03e9111028 spec: clusterID: 99854138-e415-4609-bc0b-b13f04f163b1 desiredUpdate: architecture: "" force: false image: quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 version: "" status: availableUpdates: null capabilities: enabledCapabilities: - Build - CSISnapshot - CloudControllerManager - CloudCredential - Console - DeploymentConfig - ImageRegistry - Ingress - Insights - MachineAPI - NodeTuning - OperatorLifecycleManager - OperatorLifecycleManagerV1 - Storage - baremetal - marketplace - openshift-samples knownCapabilities: - Build - CSISnapshot - CloudControllerManager - CloudCredential - Console - DeploymentConfig - ImageRegistry - Ingress - Insights - MachineAPI - NodeTuning - OperatorLifecycleManager - OperatorLifecycleManagerV1 - Storage - baremetal - marketplace - openshift-samples conditions: - lastTransitionTime: "2025-01-27T05:10:04Z" message: The update channel has not been configured. reason: NoChannel status: "False" type: RetrievedUpdates - lastTransitionTime: "2025-01-27T05:10:04Z" message: Capabilities match configured spec reason: AsExpected status: "False" type: ImplicitlyEnabledCapabilities - lastTransitionTime: "2025-01-27T05:54:48Z" message: |- Preconditions failed for payload loaded version="4.19.0-ec.1" image="quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9": Multiple precondition checks failed: * Precondition "ClusterVersionUpgradeable" failed because of "MultipleReasons": Cluster should not be upgraded between minor versions for multiple reasons: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade,UpdateInProgress * Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade. * An update is already in progress and the details are in the Progressing condition * Precondition "ClusterVersionRecommendedUpdate" failed because of "NoChannel": Configured channel is unset, so the recommended status of updating from 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest to 4.19.0-ec.1 is unknown. reason: PreconditionChecks status: "False" type: ReleaseAccepted - lastTransitionTime: "2025-01-27T05:33:20Z" message: Done applying 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest status: "True" type: Available - lastTransitionTime: "2025-01-27T05:52:39Z" status: "False" type: Failing - lastTransitionTime: "2025-01-27T05:52:31Z" message: 'Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 111 of 902 done (12% complete), waiting on etcd, kube-apiserver' reason: ClusterOperatorsUpdating status: "True" type: Progressing - lastTransitionTime: "2025-01-27T05:52:31Z" message: |- Cluster should not be upgraded between minor versions for multiple reasons: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade,UpdateInProgress * Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade. * An update is already in progress and the details are in the Progressing condition reason: MultipleReasons status: "False" type: Upgradeable - lastTransitionTime: "2025-01-27T05:54:49Z" message: 'Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade.' reason: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade status: "False" type: UpgradeableClusterOperators - lastTransitionTime: "2025-01-27T05:54:49Z" message: An update is already in progress and the details are in the Progressing condition reason: UpdateInProgress status: "True" type: UpgradeableUpgradeInProgress desired: image: registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f version: 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest history: - completionTime: null image: registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f startedTime: "2025-01-27T05:52:31Z" state: Partial verified: false version: 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest - completionTime: "2025-01-27T05:33:20Z" image: registry.build05.ci.openshift.org/ci-ln-f74xh12/release@sha256:58511cc3abf586b7b76c4389234aea79c5531888c9a89e24cc322c29e291a8e5 startedTime: "2025-01-27T05:10:04Z" state: Completed verified: false version: 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest observedGeneration: 4 versionHash: Qtwk4PYuYRA= [jianl@jianl-thinkpadt14gen4 417]$ *********************** Upgrade to 4.19 is started **************************** [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest True True 2m39s Working towards 4.19.0-ec.1: 111 of 902 done (12% complete), waiting on etcd, kube-apiserver [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade info: An upgrade is in progress. Working towards 4.19.0-ec.1: 111 of 902 done (12% complete), waiting on etcd, kube-apiserver Upgradeable=False Reason: UpdateInProgress Message: An update is already in progress and the details are in the Progressing condition warning: Cannot display available updates: Reason: NoChannel Message: The update channel has not been configured. [jianl@jianl-thinkpadt14gen4 417]$ *********************** Upgrade to 4.19 is finished **************************** [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.19.0-ec.1 True False 41s Cluster version is 4.19.0-ec.1 [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade Cluster version is 4.19.0-ec.1 Upgradeable=False Reason: UpdateInProgress Message: An update is already in progress and the details are in the Progressing condition warning: Cannot display available updates: Reason: NoChannel Message: The update channel has not been configured. [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade Cluster version is 4.19.0-ec.1 warning: Cannot display available updates: Reason: NoChannel Message: The update channel has not been configured. [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion -oyaml apiVersion: v1 items: - apiVersion: config.openshift.io/v1 kind: ClusterVersion metadata: creationTimestamp: "2025-01-27T05:09:18Z" generation: 5 name: version resourceVersion: "134541" uid: 38a7a0a7-3226-4659-844d-3e03e9111028 spec: clusterID: 99854138-e415-4609-bc0b-b13f04f163b1 desiredUpdate: architecture: "" force: false image: quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 version: "" status: availableUpdates: null capabilities: enabledCapabilities: - Build - CSISnapshot - CloudControllerManager - CloudCredential - Console - DeploymentConfig - ImageRegistry - Ingress - Insights - MachineAPI - NodeTuning - OperatorLifecycleManager - OperatorLifecycleManagerV1 - Storage - baremetal - marketplace - openshift-samples knownCapabilities: - Build - CSISnapshot - CloudControllerManager - CloudCredential - Console - DeploymentConfig - ImageRegistry - Ingress - Insights - MachineAPI - NodeTuning - OperatorLifecycleManager - OperatorLifecycleManagerV1 - Storage - baremetal - marketplace - openshift-samples conditions: - lastTransitionTime: "2025-01-27T05:10:04Z" message: The update channel has not been configured. reason: NoChannel status: "False" type: RetrievedUpdates - lastTransitionTime: "2025-01-27T05:10:04Z" message: Capabilities match configured spec reason: AsExpected status: "False" type: ImplicitlyEnabledCapabilities - lastTransitionTime: "2025-01-27T07:05:32Z" message: Payload loaded version="4.19.0-ec.1" image="quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9" architecture="amd64" reason: PayloadLoaded status: "True" type: ReleaseAccepted - lastTransitionTime: "2025-01-27T05:33:20Z" message: Done applying 4.19.0-ec.1 status: "True" type: Available - lastTransitionTime: "2025-01-27T06:53:54Z" status: "False" type: Failing - lastTransitionTime: "2025-01-27T08:15:01Z" message: Cluster version is 4.19.0-ec.1 status: "False" type: Progressing desired: image: quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 version: 4.19.0-ec.1 history: - acceptedRisks: 'Precondition "ClusterVersionRecommendedUpdate" failed because of "NoChannel": Configured channel is unset, so the recommended status of updating from 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest to 4.19.0-ec.1 is unknown.' completionTime: "2025-01-27T08:15:01Z" image: quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 startedTime: "2025-01-27T07:05:32Z" state: Completed verified: true version: 4.19.0-ec.1 - completionTime: "2025-01-27T07:00:23Z" image: registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f startedTime: "2025-01-27T05:52:31Z" state: Completed verified: false version: 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest - completionTime: "2025-01-27T05:33:20Z" image: registry.build05.ci.openshift.org/ci-ln-f74xh12/release@sha256:58511cc3abf586b7b76c4389234aea79c5531888c9a89e24cc322c29e291a8e5 startedTime: "2025-01-27T05:10:04Z" state: Completed verified: false version: 4.17.0-0.test-2025-01-27-045631-ci-ln-f74xh12-latest observedGeneration: 5 versionHash: fMWZYKQME5s= kind: List metadata: resourceVersion: "" [jianl@jianl-thinkpadt14gen4 417]$