-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.16
-
None
-
No
-
False
-
Description of problem:
periodic-ci-openshift-openshift-tests-private-release-4.16-multi-nightly-4.16-upgrade-from-stable-4.15-azure-ipi-fullyprivate-proxy-arm-f28
Running command: oc adm upgrade --to-image=registry.build02.ci.openshift.org/ci-op-y87p0c68/release@sha256:0453dcf90e6f1e6ba6b8eb197d520c95f47cb2e3906dc4d98902f10412f90ceb --allow-explicit-upgrade --force=true
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Requested update to release image registry.build02.ci.openshift.org/ci-op-y87p0c68/release@sha256:0453dcf90e6f1e6ba6b8eb197d520c95f47cb2e3906dc4d98902f10412f90ceb
Upgrading cluster to registry.build02.ci.openshift.org/ci-op-y87p0c68/release@sha256:0453dcf90e6f1e6ba6b8eb197d520c95f47cb2e3906dc4d98902f10412f90ceb gets started...
Starting the upgrade checking on 2024-06-30 18:24:46
Running command: oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.15.19 True True 4m44s Working towards 4.16.0-0.nightly-multi-2024-06-27-053432: 110 of 894 done (12% complete), waiting on etcd, kube-apiserver
Version-Release number of selected component (if applicable):
4.16.0-0.nightly-multi-2024-06-27-053432
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
# oc adm upgrade status
Unable to fetch alerts, ignoring alerts in 'Update Health': failed to get alerts from Thanos: no token is currently in use for this session
= Control Plane =
Assessment: Progressing
Target Version: 4.16.0-0.nightly-multi-2024-06-27-053432 (from 4.15.19)
Completion: 97%
Duration: 2h10m21.917016007s
Operator Status: 25 Healthy, 1 Unavailable, 7 Available but degraded
Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ci-op-y87p0c68-80996-qqdwx-master-2 Progressing Updating 4.15.19 +20m
ci-op-y87p0c68-80996-qqdwx-master-0 Completed Updated 4.16.0-0.nightly-multi-2024-06-27-053432 -
ci-op-y87p0c68-80996-qqdwx-master-1 Completed Updated 4.16.0-0.nightly-multi-2024-06-27-053432 -
= Worker Upgrade =
= Worker Pool =
Worker Pool: worker
Assessment: Completed
Completion: 100%
Worker Status: 4 Total, 4 Available, 0 Progressing, 0 Outdated, 0 Draining, 0 Excluded, 0 Degraded
Worker Pool Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ci-op-y87p0c68-80996-qqdwx-41804-k7z6m Completed Updated 4.16.0-0.nightly-multi-2024-06-27-053432 -
ci-op-y87p0c68-80996-qqdwx-worker-southcentralus1-w4dc4 Completed Updated 4.16.0-0.nightly-multi-2024-06-27-053432 -
ci-op-y87p0c68-80996-qqdwx-worker-southcentralus2-8fg97 Completed Updated 4.16.0-0.nightly-multi-2024-06-27-053432 -
ci-op-y87p0c68-80996-qqdwx-worker-southcentralus3-2kskb Completed Updated 4.16.0-0.nightly-multi-2024-06-27-053432 -
= Worker Pool =
Worker Pool: worker-pao
Assessment: Completed
Completion: 100%
Worker Status: 1 Total, 1 Available, 0 Progressing, 0 Outdated, 0 Draining, 0 Excluded, 0 Degraded
Worker Pool Node
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ci-op-y87p0c68-80996-qqdwx-worker-southcentralus1-w4dc4 Completed Updated 4.16.0-0.nightly-multi-2024-06-27-053432 -
= Update Health =
Message: Cluster Operator authentication is degraded (APIServerDeployment_UnavailablePod::OAuthServerDeployment_UnavailablePod)
Since: 43m58s
Level: Error
Impact: API Availability
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md
Resources:
clusteroperators.config.openshift.io: authentication
Description: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver ()
, OAuthServerDeploymentDegraded: 1 of 3 requested instances are unavailable for oauth-openshift.openshift-authentication ()
Message: Cluster Operator machine-config is degraded (MachineConfigDaemonFailed)
Since: 44m28s
Level: Error
Impact: API Availability
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md
Resources:
clusteroperators.config.openshift.io: machine-config
Description: Unable to apply 4.16.0-0.nightly-multi-2024-06-27-053432: error during waitForDaemonsetRollout: [context deadline exceeded, daemonset machine-config-daemon is not ready. status: (desired: 7, updated: 7, ready: 6, unavailable: 1)]
Message: Cluster Operator openshift-apiserver is degraded (APIServerDeployment_UnavailablePod)
Since: 44m32s
Level: Error
Impact: API Availability
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md
Resources:
clusteroperators.config.openshift.io: openshift-apiserver
Description: APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-apiserver ()
Message: Cluster Operator etcd is degraded (EtcdCertSignerController_Error::EtcdEndpoints_ErrorUpdatingEtcdEndpoints::EtcdMembers_UnhealthyMembers::NodeController_MasterNodesReady)
Since: 1h8m47s
Level: Error
Impact: API Availability
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md
Resources:
clusteroperators.config.openshift.io: etcd
Description: EtcdCertSignerControllerDegraded: EtcdCertSignerController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:1612627219685692483 name:"ci-op-y87p0c68-80996-qqdwx-master-2" peerURLs:"https://10.0.0.7:2380" clientURLs:"https://10.0.0.7:2379" Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.0.7:2379]: context deadline exceeded} {Member:ID:7859641474667542729 name:"ci-op-y87p0c68-80996-qqdwx-master-0" peerURLs:"https://10.0.0.8:2380" clientURLs:"https://10.0.0.8:2379" Healthy:true Took:1.638123ms Error:<nil>} {Member:ID:16394202218457999149 name:"ci-op-y87p0c68-80996-qqdwx-master-1" peerURLs:"https://10.0.0.6:2380" clientURLs:"https://10.0.0.6:2379" Healthy:true Took:2.867525ms Error:<nil>}]
, EtcdEndpointsDegraded: EtcdEndpointsController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:1612627219685692483 name:"ci-op-y87p0c68-80996-qqdwx-master-2" peerURLs:"https://10.0.0.7:2380" clientURLs:"https://10.0.0.7:2379" Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.0.7:2379]: context deadline exceeded} {Member:ID:7859641474667542729 name:"ci-op-y87p0c68-80996-qqdwx-master-0" peerURLs:"https://10.0.0.8:2380" clientURLs:"https://10.0.0.8:2379" Healthy:true Took:3.586207ms Error:<nil>} {Member:ID:16394202218457999149 name:"ci-op-y87p0c68-80996-qqdwx-master-1" peerURLs:"https://10.0.0.6:2380" clientURLs:"https://10.0.0.6:2379" Healthy:true Took:1.774003ms Error:<nil>}]
, EtcdMembersDegraded: 2 of 3 members are available, ci-op-y87p0c68-80996-qqdwx-master-2 is unhealthy
, NodeControllerDegraded: The master nodes not ready: node "ci-op-y87p0c68-80996-qqdwx-master-2" not ready since 2024-06-30 19:24:33 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
Message: Cluster Operator kube-apiserver is degraded (NodeController_MasterNodesReady)
Since: 1h8m50s
Level: Error
Impact: API Availability
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md
Resources:
clusteroperators.config.openshift.io: kube-apiserver
Description: NodeControllerDegraded: The master nodes not ready: node "ci-op-y87p0c68-80996-qqdwx-master-2" not ready since 2024-06-30 19:24:33 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
Message: Cluster Operator kube-controller-manager is degraded (NodeController_MasterNodesReady)
Since: 1h8m50s
Level: Error
Impact: API Availability
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md
Resources:
clusteroperators.config.openshift.io: kube-controller-manager
Description: NodeControllerDegraded: The master nodes not ready: node "ci-op-y87p0c68-80996-qqdwx-master-2" not ready since 2024-06-30 19:24:33 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
Message: Cluster Operator kube-scheduler is degraded (NodeController_MasterNodesReady)
Since: 1h8m50s
Level: Error
Impact: API Availability
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md
Resources:
clusteroperators.config.openshift.io: kube-scheduler
Description: NodeControllerDegraded: The master nodes not ready: node "ci-op-y87p0c68-80996-qqdwx-master-2" not ready since 2024-06-30 19:24:33 +0000 UTC because NodeStatusUnknown (Kubelet stopped posting node status.)
Message: Cluster Operator control-plane-machine-set is unavailable (UnavailableReplicas)
Since: 19m45s
Level: Warning
Impact: API Availability
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDown.md
Resources:
clusteroperators.config.openshift.io: control-plane-machine-set
Description: Missing 1 available replica(s)
Message: Cluster Version version is failing to proceed with the update (ClusterOperatorsDegraded)
Since: 28m41s
Level: Warning
Impact: Update Stalled
Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md
Resources:
clusterversions.config.openshift.io: version
Description: Cluster operators etcd, kube-apiserver are degraded
Expected results:
Upgrade should pass
Additional info:
