-
Bug
-
Resolution: Done
-
Major
-
4.9
-
False
-
False
-
NEW
-
NEW
-
undefined
-
Issue is with below builds :
4.9.0-0.nightly-ppc64le-2021-08-17-145337
4.9.0-0.nightly-ppc64le-2021-08-19-120135
on bastion :
- lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 8
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Model: 2.3 (pvr 004e 0203)
Model name: POWER9 (architected), altivec supported
Hypervisor vendor: pHyp
Virtualization type: para
L1d cache: 32K
L1i cache: 32K
NUMA node0 CPU(s): 0-7
Physical sockets: 2
Physical chips: 1
Physical cores/chip: 10
[core@master-0 ~]$ lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 8
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Model: 2.3 (pvr 004e 0203)
Model name: POWER9 (architected), altivec supported
Hypervisor vendor: pHyp
Virtualization type: para
L1d cache: 32K
L1i cache: 32K
NUMA node0 CPU(s): 0-7
No workload was deployed on the cluster.
- oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False False True 4d5h OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.pravin-49proxy.redhat.com/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...
baremetal 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
cloud-controller-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
cloud-credential 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
cluster-autoscaler 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
config-operator 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
console 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False False False 4d6h RouteHealthAvailable: console route is not admitted
csi-snapshot-controller 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True True False 3d21h Progressing: Waiting for Deployment to deploy csi-snapshot-controller pods
dns 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
etcd 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
image-registry 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d15h
ingress 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
insights 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
kube-apiserver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False True 6d16h NodeControllerDegraded: The master nodes not ready: node "master-2" not ready since 2021-08-23 09:51:21 +0000 UTC because KubeletNotReady (container runtime is down)...
kube-controller-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False True 6d16h StaticPodsDegraded: pod/kube-controller-manager-master-0 container "kube-controller-manager" is waiting: CreateContainerError: open /var/run/containers/storage/overlay-layers/.tmp-mountpoints.json632783936: no space left on device...
kube-scheduler 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False True 6d16h StaticPodsDegraded: pod/openshift-kube-scheduler-master-0 container "kube-scheduler" is waiting: CreateContainerError: open /var/run/containers/storage/overlay-layers/.tmp-mountpoints.json574670570: no space left on device...
kube-storage-version-migrator 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 4d10h
machine-api 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
machine-approver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
machine-config 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 4d5h
marketplace 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
monitoring 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False True True 4d22h Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
network 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True True True 6d16h DaemonSet "openshift-sdn/sdn" rollout is not making progress - last change 2021-08-20T11:46:25Z
node-tuning 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
openshift-apiserver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False False False 18h APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.84:8443/apis/apps.openshift.io/v1: bad status from https://10.129.0.84:8443/apis/apps.openshift.io/v1: 401...
openshift-controller-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
openshift-samples 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
operator-lifecycle-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
operator-lifecycle-manager-catalog 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
operator-lifecycle-manager-packageserver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False True False 3d21h ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: InstallCheckFailed, message: install timeout
service-ca 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
storage 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
- oc get nodes
NAME STATUS ROLES AGE VERSION
master-0 Ready master 6d17h v1.22.0-rc.0+3dfed96
master-1 Ready master 6d17h v1.22.0-rc.0+3dfed96
master-2 Ready master 6d17h v1.22.0-rc.0+3dfed96
worker-0 Ready worker 6d17h v1.22.0-rc.0+3dfed96
worker-1 Ready worker 6d17h v1.22.0-rc.0+3dfed96
- oc adm top nodes
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
- oc get pods -A -o wide | grep -v "Running" | grep -v "Completed"
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nfs-provisioner nfs-client-provisioner-84766f5b6d-6pscv 0/1 CreateContainerError 0 (4d19h ago) 6d18h 10.128.2.9 worker-1 <none> <none>
openshift-apiserver apiserver-9fc69858-9tkhg 1/2 CreateContainerError 11 (43h ago) 6d17h 10.128.0.47 master-0 <none> <none>
openshift-apiserver apiserver-9fc69858-p9x8t 1/2 CreateContainerError 6 (25h ago) 6d18h 10.130.0.32 master-2 <none> <none>
openshift-authentication-operator authentication-operator-656f485cc6-8rtk8 0/1 CreateContainerError 6 (3d23h ago) 4d8h 10.130.0.62 master-2 <none> <none>
openshift-authentication oauth-openshift-54bcbd5487-nhkcv 0/1 CreateContainerError 1 (44h ago) 4d8h 10.130.0.70 master-2 <none> <none>
openshift-cluster-storage-operator csi-snapshot-controller-74cbcb446f-cf572 0/1 CreateContainerError 6 (3d23h ago) 4d8h 10.130.0.60 master-2 <none> <none>
openshift-cluster-storage-operator csi-snapshot-controller-74cbcb446f-hz6wr 0/1 CreateContainerError 80 (3d23h ago) 6d18h 10.128.0.4 master-0 <none> <none>
openshift-cluster-version cluster-version-operator-cdd5f5c56-t6d2j 0/1 CreateContainerError 7 (43h ago) 6d18h 9.47.87.104 master-0 <none> <none>
openshift-config-operator openshift-config-operator-58684b8cc5-7k658 0/1 CreateContainerError 6 (43h ago) 4d8h 10.128.0.69 master-0 <none> <none>
openshift-console-operator console-operator-7b4ddbcf8d-2h4vw 0/1 CreateContainerError 53 (4d8h ago) 6d18h 10.130.0.22 master-2 <none> <none>
openshift-console console-649f685c96-g7h96 0/1 CreateContainerError 4 (44h ago) 6d18h 10.130.0.24 master-2 <none> <none>
openshift-console console-649f685c96-jzgx7 0/1 CreateContainerError 3 (43h ago) 6d18h 10.128.0.39 master-0 <none> <none>
openshift-image-registry image-pruner-27156960--1-pwn2z 0/1 ContainerCreating 0 4d10h <none> worker-1 <none> <none>
openshift-kube-apiserver-operator kube-apiserver-operator-54cbc7b859-7jg5w 0/1 CreateContainerError 2 (25h ago) 4d8h 10.130.0.68 master-2 <none> <none>
openshift-kube-apiserver kube-apiserver-master-0 4/5 CreateContainerError 13 (43h ago) 5d18h 9.47.87.104 master-0 <none> <none>
openshift-kube-apiserver kube-apiserver-master-2 4/5 CreateContainerError 6 (4d7h ago) 5d18h 9.47.87.106 master-2 <none> <none>
openshift-kube-controller-manager kube-controller-manager-master-0 3/4 CreateContainerError 88 (3d23h ago) 6d18h 9.47.87.104 master-0 <none> <none>
openshift-kube-controller-manager kube-controller-manager-master-1 2/4 CreateContainerError 108 (4d ago) 6d18h 9.47.87.110 master-1 <none> <none>
openshift-kube-controller-manager kube-controller-manager-master-2 3/4 CreateContainerError 74 (4d7h ago) 6d18h 9.47.87.106 master-2 <none> <none>
openshift-kube-scheduler openshift-kube-scheduler-master-0 2/3 CreateContainerError 67 (2d10h ago) 6d18h 9.47.87.104 master-0 <none> <none>
openshift-kube-scheduler openshift-kube-scheduler-master-1 2/3 CreateContainerError 59 (4d3h ago) 6d18h 9.47.87.110 master-1 <none> <none>
openshift-marketplace certified-operators-k7r4p 0/1 ImageInspectError 0 (5d1h ago) 5d2h 10.128.3.108 worker-1 <none> <none>
openshift-marketplace certified-operators-l7gv2 0/1 ImageInspectError 0 5d1h 10.128.3.126 worker-1 <none> <none>
openshift-marketplace community-operators-656xf 0/1 ImageInspectError 0 (5d1h ago) 5d1h 10.128.3.118 worker-1 <none> <none>
openshift-marketplace community-operators-znvc2 0/1 ImageInspectError 0 5d1h 10.128.3.127 worker-1 <none> <none>
openshift-marketplace marketplace-operator-54c6798b65-77vwn 0/1 CreateContainerError 6 (3d23h ago) 4d8h 10.128.0.62 master-0 <none> <none>
openshift-marketplace redhat-marketplace-h4rdg 0/1 ContainerCreating 0 5d1h <none> worker-1 <none> <none>
openshift-marketplace redhat-marketplace-zk6tn 0/1 ImageInspectError 0 (5d1h ago) 6d14h 10.128.2.87 worker-1 <none> <none>
openshift-marketplace redhat-operators-9k5dj 0/1 ImageInspectError 0 (5d1h ago) 5d1h 10.128.3.116 worker-1 <none> <none>
openshift-marketplace redhat-operators-9ldp7 0/1 ImageInspectError 0 5d1h 10.128.3.124 worker-1 <none> <none>
openshift-monitoring alertmanager-main-1 0/5 ContainerCreating 0 4d7h <none> worker-1 <none> <none>
openshift-monitoring grafana-8657cf6659-dx97s 0/2 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-monitoring kube-state-metrics-7b6988cf78-7x8sw 0/3 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-monitoring openshift-state-metrics-5785f48976-nlzv5 0/3 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-monitoring prometheus-adapter-5f6ff5849b-r79hm 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
openshift-monitoring prometheus-adapter-6bcc5ddf97-krzsn 0/1 Pending 0 3d23h <none> <none> <none> <none>
openshift-monitoring prometheus-adapter-6bcc5ddf97-qq7df 0/1 ContainerCreating 0 3d23h <none> worker-1 <none> <none>
openshift-monitoring prometheus-k8s-0 6/7 CreateContainerError 0 (3d14h ago) 4d7h 10.131.0.249 worker-0 <none> <none>
openshift-monitoring prometheus-k8s-1 0/7 Init:0/1 0 4d7h <none> worker-1 <none> <none>
openshift-monitoring telemeter-client-7b954d746b-fspb5 0/3 ContainerCreating 0 4d9h <none> worker-1 <none> <none>
openshift-network-diagnostics network-check-source-dfbc58d6f-mb2b5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-oauth-apiserver apiserver-6d5b547944-r2njv 0/1 CreateContainerError 3 (45h ago) 4d8h 10.129.0.85 master-1 <none> <none>
openshift-oauth-apiserver apiserver-6d5b547944-zqnh6 0/1 CreateContainerError 7 (4d7h ago) 6d18h 10.130.0.9 master-2 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156585--1-7gnj8 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156600--1-bfwnf 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156615--1-hvhzf 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156630--1-bm8dh 0/1 ContainerCreating 0 4d16h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156645--1-nqcsj 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156660--1-gckl4 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156675--1-d2hvq 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156690--1-n6pbn 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156705--1-flx77 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156720--1-ml56f 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156735--1-6brkc 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156750--1-9q4k2 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156765--1-kwch8 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156780--1-8vkj5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156795--1-qg8fw 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156810--1-hz9mm 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156825--1-w2wvk 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156840--1-mv2fd 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156855--1-fpgrn 0/1 ContainerCreating 0 4d12h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156870--1-6c7ck 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156885--1-z58sv 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156900--1-q885s 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156915--1-78mhb 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156930--1-898zx 0/1 ContainerCreating 0 4d11h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156945--1-5grj9 0/1 ContainerCreating 0 4d11h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156960--1-q77nv 0/1 ContainerCreating 0 4d10h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156975--1-tqr9j 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27156990--1-87mdj 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157005--1-rcr7m 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157020--1-x6jdv 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157035--1-5mhjc 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157050--1-8nbjz 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157065--1-xfvms 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157080--1-g5f9q 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157095--1-tcch5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157110--1-jh8x5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157125--1-lfcl4 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157140--1-pfb9r 0/1 ContainerCreating 0 4d7h <none> worker-1 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157185--1-c99ph 0/1 ContainerCreating 0 4d7h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157200--1-vc9kr 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157215--1-7qxjj 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157230--1-bmr9s 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157245--1-8sjsd 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157260--1-vrchv 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157275--1-tdq9c 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157290--1-v782d 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157305--1-5d5zb 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157320--1-jwh24 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157335--1-gltgf 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157350--1-jbmhc 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157365--1-jbmn8 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager collect-profiles-27157665--1-4spth 0/1 ContainerCreating 0 3d23h <none> worker-0 <none> <none>
openshift-operator-lifecycle-manager olm-operator-75dcc9cb6-8nt5n 0/1 CreateContainerError 4 (43h ago) 4d8h 10.128.0.65 master-0 <none> <none>
openshift-operator-lifecycle-manager package-server-manager-7b7646568c-gkx7k 0/1 CreateContainerError 6 (4d7h ago) 4d8h 10.130.0.55 master-2 <none> <none>
openshift-operator-lifecycle-manager packageserver-76694fb586-9pph6 0/1 CreateContainerError 7 (43h ago) 4d8h 10.128.0.68 master-0 <none> <none>
- blocks
-
MULTIARCH-1648 "no space left on device" issue is seen on latest 4.8 builds
- Closed
- external trackers