-
Bug
-
Resolution: Done
-
Minor
-
4.13.0
-
No
-
Rejected
-
False
-
Description of problem:
On attempting to upgrade from 4.12.3 to RHCOS 413.92.202302081904-0,I see master mcp in degraded state with the below message: =================== unexpected on-disk state validating against rendered-master-733a7422eec1f1e0d3de151a31b7a054: expected target osImageURL "registry.build05.ci.openshift.org/ci-ln-5f80vhb/stable@sha256:4b5fb23df6b360e41da18e001d56e4fb5e9e575318a0b9000a1a316c74c9515d", have "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50fa4c3f33d7fb8a3f8db93636937cb888a55718fe3833f339cbcc55dc6d0e3a" ("3abc4116c09e75764c30cc14918dfc343d37b9368b5d10cf2a9f5bd278de046b"); possible root cause: error: Child process exited with code 1
Version-Release number of selected component (if applicable):
4.12.3 to RHCOS 413.92.202302081904-0
How reproducible:
100%
Steps to Reproduce:
1. Run ocp upgrade from 4.12.3 to to RHCOS 413.92.202302081904-0 2. 3.
Actual results:
cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get nodes NAME STATUS ROLES AGE VERSION cnv-qe-infra-29.cnvqe2.lab.eng.rdu2.redhat.com Ready control-plane,master 25h v1.25.4+a34b9e9 cnv-qe-infra-30.cnvqe2.lab.eng.rdu2.redhat.com Ready control-plane,master 25h v1.25.4+a34b9e9 cnv-qe-infra-31.cnvqe2.lab.eng.rdu2.redhat.com Ready,SchedulingDisabled control-plane,master 25h v1.25.4+a34b9e9 cnv-qe-infra-32.cnvqe2.lab.eng.rdu2.redhat.com NotReady,SchedulingDisabled worker 25h v1.25.4+a34b9e9 cnv-qe-infra-33.cnvqe2.lab.eng.rdu2.redhat.com Ready worker 25h v1.25.4+a34b9e9 cnv-qe-infra-34.cnvqe2.lab.eng.rdu2.redhat.com Ready worker 25h v1.25.4+a34b9e9 cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get mcp NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-1ff7b81307234c62d951a7ef88d7fbc9 False True True 3 0 0 1 25h worker rendered-worker-c344a2d515cc2c21644075a59e573a7d False True False 3 0 0 0 25h cnv-qe-jenkins@cnv-qe-infra-01:~$ cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True True True 11m APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver ()... cloud-controller-manager 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h cloud-credential 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h cluster-autoscaler 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h config-operator 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h console 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h control-plane-machine-set 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h csi-snapshot-controller 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h dns 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True True False 26h DNS "default" reports Progressing=True: "Have 5 available node-resolver pods, want 6." etcd 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h image-registry 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True True False 24h Progressing: The registry is ready... ingress 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h insights 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h kube-apiserver 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h kube-controller-manager 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h kube-scheduler 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h kube-storage-version-migrator 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h machine-api 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h machine-approver 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h machine-config 4.12.3 False True True 20h Cluster not available for [{operator 4.12.3}]: failed to apply machine config daemon manifests: error during waitForDaemonsetRollout: [timed out waiting for the condition, daemonset machine-config-daemon is not ready. status: (desired: 6, updated: 6, ready: 5, unavailable: 1)] marketplace 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h monitoring 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h network 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True True False 26h DaemonSet "/openshift-multus/multus" is not available (awaiting 1 nodes)... node-tuning 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 21h openshift-apiserver 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False True 25h APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver () openshift-controller-manager 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True True False 25h Progressing: deployment/controller-manager: updated replicas is 1, desired replicas is 3... openshift-samples 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 21h operator-lifecycle-manager 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h operator-lifecycle-manager-catalog 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h operator-lifecycle-manager-packageserver 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 25h service-ca 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h storage 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest True False False 26h cnv-qe-jenkins@cnv-qe-infra-01:~$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.12.3 True True 21h Unable to apply 4.13.0-0.ci.test-2023-02-10-134923-ci-ln-5f80vhb-latest: wait has exceeded 40 minutes for these operators: authentication, openshift-apiserver cnv-qe-jenkins@cnv-qe-infra-01:~$
Expected results:
Successful upgrade
Additional info:
- relates to
-
COS-1926 Move RHCOS to RHEL 9.2 in OCP 4.13
- Closed