Create 4.18 image: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/release-openshift-origin-installer-launch-aws-modern/1883694520944562176 oc image info registry.build05.ci.openshift.org/ci-ln-4d1ld22/release:latest | grep 'Digest:' | tr -d ' ' | cut -d ':' -f2- [jianl@jianl-thinkpadt14gen4 417]$ oc image info registry.build05.ci.openshift.org/ci-ln-4d1ld22/release:latest | grep 'Digest:' | tr -d ' ' | cut -d ':' -f2- sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.17.0-0.nightly-2025-01-25-055503 True False 33m Cluster version is 4.17.0-0.nightly-2025-01-25-055503 [jianl@jianl-thinkpadt14gen4 417]$ *********************Upgrade to 4.18************************* [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f error: cannot refresh available updates: Reason: VersionNotFound Message: Unable to retrieve available updates: currently reconciling cluster version 4.17.0-0.nightly-2025-01-25-055503 not found in the "stable-4.17" channel specify --allow-explicit-upgrade to continue with the update. [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway Requested update to release image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f --allow-explicit-upgrade --force warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Requested update to release image registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.17.0-0.nightly-2025-01-25-055503 True True 4s Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 11 of 902 done (1% complete) [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade info: An upgrade is in progress. Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 6 of 902 done (0% complete) Upgradeable=False Reason: UpdateInProgress Message: An update is already in progress and the details are in the Progressing condition Upstream is unset, so the cluster will use an appropriate default. Channel: stable-4.17 warning: Cannot display available updates: Reason: VersionNotFound Message: Unable to retrieve available updates: currently reconciling cluster version 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest not found in the "stable-4.17" channel [jianl@jianl-thinkpadt14gen4 417]$ ******************Upgrade to 4.19.0-ec.1****************** [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 error: cannot refresh available updates: Reason: VersionNotFound Message: Unable to retrieve available updates: currently reconciling cluster version 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest not found in the "stable-4.17" channel specify --allow-explicit-upgrade to continue with the update. [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway error: the cluster is already upgrading: Reason: ClusterOperatorsUpdating Message: Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 111 of 902 done (12% complete), waiting on etcd, kube-apiserver If you want to upgrade anyway, use --allow-upgrade-with-warnings. [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 --allow-explicit-upgrade --allow-upgrade-with-warnings warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway warning: --allow-upgrade-with-warnings is bypassing: the cluster is already upgrading: Reason: ClusterOperatorsUpdating Message: Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 111 of 902 done (12% complete), waiting on etcd, kube-apiserver Requested update to release image quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 [jianl@jianl-thinkpadt14gen4 417]$ ****************** conditions ****************** [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion version -o json | jq .status.conditions [ { "lastTransitionTime": "2025-01-27T02:05:44Z", "message": "Unable to retrieve available updates: currently reconciling cluster version 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest not found in the \"stable-4.17\" channel", "reason": "VersionNotFound", "status": "False", "type": "RetrievedUpdates" }, { "lastTransitionTime": "2025-01-27T02:05:44Z", "message": "Capabilities match configured spec", "reason": "AsExpected", "status": "False", "type": "ImplicitlyEnabledCapabilities" }, { "lastTransitionTime": "2025-01-27T03:13:12Z", "message": "Preconditions failed for payload loaded version=\"4.19.0-ec.1\" image=\"quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9\": Multiple precondition checks failed:\n* Precondition \"ClusterVersionUpgradeable\" failed because of \"MultipleReasons\": Cluster should not be upgraded between minor versions for multiple reasons: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade,UpdateInProgress\n* Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade.\n* An update is already in progress and the details are in the Progressing condition\n* Precondition \"ClusterVersionRecommendedUpdate\" failed because of \"UnknownUpdate\": RetrievedUpdates=False (VersionNotFound), so the recommended status of updating from 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest to 4.19.0-ec.1 is unknown.", "reason": "PreconditionChecks", "status": "False", "type": "ReleaseAccepted" }, { "lastTransitionTime": "2025-01-27T02:31:29Z", "message": "Done applying 4.17.0-0.nightly-2025-01-25-055503", "status": "True", "type": "Available" }, { "lastTransitionTime": "2025-01-27T03:11:54Z", "status": "False", "type": "Failing" }, { "lastTransitionTime": "2025-01-27T03:11:32Z", "message": "Working towards 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest: 233 of 902 done (25% complete), waiting on machine-api", "reason": "ClusterOperatorUpdating", "status": "True", "type": "Progressing" }, { "lastTransitionTime": "2025-01-27T03:11:39Z", "message": "Cluster should not be upgraded between minor versions for multiple reasons: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade,UpdateInProgress\n* Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade.\n* An update is already in progress and the details are in the Progressing condition", "reason": "MultipleReasons", "status": "False", "type": "Upgradeable" }, { "lastTransitionTime": "2025-01-27T03:13:13Z", "message": "Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade.", "reason": "KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade", "status": "False", "type": "UpgradeableClusterOperators" }, { "lastTransitionTime": "2025-01-27T03:13:13Z", "message": "An update is already in progress and the details are in the Progressing condition", "reason": "UpdateInProgress", "status": "True", "type": "UpgradeableUpgradeInProgress" } ] [jianl@jianl-thinkpadt14gen4 417]$ ****************** HISTORY ****************** [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion version -o json | jq .status.history [ { "completionTime": null, "image": "registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f", "startedTime": "2025-01-27T03:11:32Z", "state": "Partial", "verified": false, "version": "4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest" }, { "completionTime": "2025-01-27T02:31:29Z", "image": "registry.ci.openshift.org/ocp/release@sha256:cc95bff0a8ed2d0cb18f8b371f8d6dd536dcc70f76844ef7c33cfb391440a588", "startedTime": "2025-01-27T02:05:44Z", "state": "Completed", "verified": false, "version": "4.17.0-0.nightly-2025-01-25-055503" } ] [jianl@jianl-thinkpadt14gen4 417]$ ******************************* status ************************************** [jianl@jianl-thinkpadt14gen4 417]$ export OC_ENABLE_CMD_UPGRADE_STATUS=true [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ [jianl@jianl-thinkpadt14gen4 417]$ oc adm upgrade status Unable to fetch alerts, ignoring alerts in 'Update Health': failed to get alerts from Thanos: no token is currently in use for this session = Control Plane = Assessment: Progressing Target Version: 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest (from 4.17.0-0.nightly-2025-01-25-055503) Updating: monitoring Completion: 85% (28 operators updated, 1 updating, 4 waiting) Duration: 29m (Est. Time Remaining: 40m) Operator Health: 33 Healthy Control Plane Nodes NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-19-150.us-east-2.compute.internal Outdated Pending 4.17.0-0.nightly-2025-01-25-055503 ? ip-10-0-52-88.us-east-2.compute.internal Outdated Pending 4.17.0-0.nightly-2025-01-25-055503 ? ip-10-0-80-46.us-east-2.compute.internal Outdated Pending 4.17.0-0.nightly-2025-01-25-055503 ? = Worker Upgrade = WORKER POOL ASSESSMENT COMPLETION STATUS worker Pending 0% (0/3) 3 Available, 0 Progressing, 0 Draining Worker Pool Nodes: worker NAME ASSESSMENT PHASE VERSION EST MESSAGE ip-10-0-26-81.us-east-2.compute.internal Outdated Pending 4.17.0-0.nightly-2025-01-25-055503 ? ip-10-0-50-95.us-east-2.compute.internal Outdated Pending 4.17.0-0.nightly-2025-01-25-055503 ? ip-10-0-87-145.us-east-2.compute.internal Outdated Pending 4.17.0-0.nightly-2025-01-25-055503 ? = Update Health = SINCE LEVEL IMPACT MESSAGE 29m11s Info None Update is proceeding well [jianl@jianl-thinkpadt14gen4 417]$ ******************ON UI****************** Warning alert:This cluster should not be updated to the next minor version. Cluster should not be upgraded between minor versions for multiple reasons: KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade,UpdateInProgress Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade. An update is already in progress and the details are in the Progressing condition View ClusterOperators When upgrade to 4.18 finished, upgrade to 4.19 is starting: [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest True True 14m Working towards 4.19.0-ec.1: 133 of 902 done (14% complete), waiting on kube-controller-manager, kube-scheduler [jianl@jianl-thinkpadt14gen4 417]$ And console UI is working again [jianl@jianl-thinkpadt14gen4 417]$ oc get clusterversion version -oyaml apiVersion: config.openshift.io/v1 kind: ClusterVersion metadata: creationTimestamp: "2025-01-27T02:05:25Z" generation: 5 name: version resourceVersion: "94135" uid: 8c178243-82c1-4592-b9fc-14da49f4528b spec: channel: stable-4.17 clusterID: c6f11b01-1d7c-4f4d-8c16-3363dc8d0e85 desiredUpdate: architecture: "" force: false image: quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 version: "" status: availableUpdates: null capabilities: enabledCapabilities: - Build - CSISnapshot - CloudControllerManager - CloudCredential - Console - DeploymentConfig - ImageRegistry - Ingress - Insights - MachineAPI - NodeTuning - OperatorLifecycleManager - OperatorLifecycleManagerV1 - Storage - baremetal - marketplace - openshift-samples knownCapabilities: - Build - CSISnapshot - CloudControllerManager - CloudCredential - Console - DeploymentConfig - ImageRegistry - Ingress - Insights - MachineAPI - NodeTuning - OperatorLifecycleManager - OperatorLifecycleManagerV1 - Storage - baremetal - marketplace - openshift-samples conditions: - lastTransitionTime: "2025-01-27T02:05:44Z" message: 'Unable to retrieve available updates: currently reconciling cluster version 4.19.0-ec.1 not found in the "stable-4.17" channel' reason: VersionNotFound status: "False" type: RetrievedUpdates - lastTransitionTime: "2025-01-27T02:05:44Z" message: Capabilities match configured spec reason: AsExpected status: "False" type: ImplicitlyEnabledCapabilities - lastTransitionTime: "2025-01-27T04:21:16Z" message: Payload loaded version="4.19.0-ec.1" image="quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9" architecture="amd64" reason: PayloadLoaded status: "True" type: ReleaseAccepted - lastTransitionTime: "2025-01-27T02:31:29Z" message: Done applying 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest status: "True" type: Available - lastTransitionTime: "2025-01-27T04:03:57Z" status: "False" type: Failing - lastTransitionTime: "2025-01-27T04:21:16Z" message: 'Working towards 4.19.0-ec.1: 133 of 902 done (14% complete), waiting on kube-controller-manager, kube-scheduler' reason: ClusterOperatorsUpdating status: "True" type: Progressing - lastTransitionTime: "2025-01-27T04:21:16Z" message: An update is already in progress and the details are in the Progressing condition reason: UpdateInProgress status: "False" type: Upgradeable desired: image: quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 version: 4.19.0-ec.1 history: - acceptedRisks: 'Precondition "ClusterVersionRecommendedUpdate" failed because of "UnknownUpdate": RetrievedUpdates=False (VersionNotFound), so the recommended status of updating from 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest to 4.19.0-ec.1 is unknown.' completionTime: null image: quay.io/openshift-release-dev/ocp-release@sha256:aa3e0a3a94babd90535f8298ab274b51a9bce6045dda8c3c8cd742bc59f0e2d9 startedTime: "2025-01-27T04:21:16Z" state: Partial verified: true version: 4.19.0-ec.1 - completionTime: "2025-01-27T04:18:07Z" image: registry.build05.ci.openshift.org/ci-ln-4d1ld22/release@sha256:a31b94abe725c462b444d0524dbf4ed4696d9d8175989c57e376eaac9d64854f startedTime: "2025-01-27T03:11:32Z" state: Completed verified: false version: 4.18.0-0.test-2025-01-27-021124-ci-ln-4d1ld22-latest - completionTime: "2025-01-27T02:31:29Z" image: registry.ci.openshift.org/ocp/release@sha256:cc95bff0a8ed2d0cb18f8b371f8d6dd536dcc70f76844ef7c33cfb391440a588 startedTime: "2025-01-27T02:05:44Z" state: Completed verified: false version: 4.17.0-0.nightly-2025-01-25-055503 observedGeneration: 5 versionHash: fMWZYKQME5s= [jianl@jianl-thinkpadt14gen4 417]$