Uploaded image for project: 'Multiple Architecture Enablement'
  1. Multiple Architecture Enablement
  2. MULTIARCH-1605

"no space left on device" issue is seen on latest 4.9 builds

XMLWordPrintable

    • False
    • False
    • NEW
    • NEW
    • undefined

      Issue is with below builds :
      4.9.0-0.nightly-ppc64le-2021-08-17-145337
      4.9.0-0.nightly-ppc64le-2021-08-19-120135

      on bastion :

      1. lscpu
        Architecture: ppc64le
        Byte Order: Little Endian
        CPU(s): 8
        On-line CPU(s) list: 0-7
        Thread(s) per core: 8
        Core(s) per socket: 1
        Socket(s): 1
        NUMA node(s): 1
        Model: 2.3 (pvr 004e 0203)
        Model name: POWER9 (architected), altivec supported
        Hypervisor vendor: pHyp
        Virtualization type: para
        L1d cache: 32K
        L1i cache: 32K
        NUMA node0 CPU(s): 0-7
        Physical sockets: 2
        Physical chips: 1
        Physical cores/chip: 10

      [core@master-0 ~]$ lscpu
      Architecture: ppc64le
      Byte Order: Little Endian
      CPU(s): 8
      On-line CPU(s) list: 0-7
      Thread(s) per core: 8
      Core(s) per socket: 1
      Socket(s): 1
      NUMA node(s): 1
      Model: 2.3 (pvr 004e 0203)
      Model name: POWER9 (architected), altivec supported
      Hypervisor vendor: pHyp
      Virtualization type: para
      L1d cache: 32K
      L1i cache: 32K
      NUMA node0 CPU(s): 0-7

      No workload was deployed on the cluster.

      1. oc get co
        NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
        authentication 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False False True 4d5h OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.pravin-49proxy.redhat.com/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...
        baremetal 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        cloud-controller-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        cloud-credential 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        cluster-autoscaler 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        config-operator 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        console 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False False False 4d6h RouteHealthAvailable: console route is not admitted
        csi-snapshot-controller 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True True False 3d21h Progressing: Waiting for Deployment to deploy csi-snapshot-controller pods
        dns 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        etcd 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        image-registry 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d15h
        ingress 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        insights 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        kube-apiserver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False True 6d16h NodeControllerDegraded: The master nodes not ready: node "master-2" not ready since 2021-08-23 09:51:21 +0000 UTC because KubeletNotReady (container runtime is down)...
        kube-controller-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False True 6d16h StaticPodsDegraded: pod/kube-controller-manager-master-0 container "kube-controller-manager" is waiting: CreateContainerError: open /var/run/containers/storage/overlay-layers/.tmp-mountpoints.json632783936: no space left on device...
        kube-scheduler 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False True 6d16h StaticPodsDegraded: pod/openshift-kube-scheduler-master-0 container "kube-scheduler" is waiting: CreateContainerError: open /var/run/containers/storage/overlay-layers/.tmp-mountpoints.json574670570: no space left on device...
        kube-storage-version-migrator 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 4d10h
        machine-api 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        machine-approver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        machine-config 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 4d5h
        marketplace 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        monitoring 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False True True 4d22h Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
        network 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True True True 6d16h DaemonSet "openshift-sdn/sdn" rollout is not making progress - last change 2021-08-20T11:46:25Z
        node-tuning 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        openshift-apiserver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False False False 18h APIServicesAvailable: apiservices.apiregistration.k8s.io/v1.apps.openshift.io: not available: failing or missing response from https://10.129.0.84:8443/apis/apps.openshift.io/v1: bad status from https://10.129.0.84:8443/apis/apps.openshift.io/v1: 401...
        openshift-controller-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        openshift-samples 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        operator-lifecycle-manager 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        operator-lifecycle-manager-catalog 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        operator-lifecycle-manager-packageserver 4.9.0-0.nightly-ppc64le-2021-08-17-145337 False True False 3d21h ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: InstallCheckFailed, message: install timeout
        service-ca 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
        storage 4.9.0-0.nightly-ppc64le-2021-08-17-145337 True False False 6d16h
      1. oc get nodes
        NAME STATUS ROLES AGE VERSION
        master-0 Ready master 6d17h v1.22.0-rc.0+3dfed96
        master-1 Ready master 6d17h v1.22.0-rc.0+3dfed96
        master-2 Ready master 6d17h v1.22.0-rc.0+3dfed96
        worker-0 Ready worker 6d17h v1.22.0-rc.0+3dfed96
        worker-1 Ready worker 6d17h v1.22.0-rc.0+3dfed96
      1. oc adm top nodes
        Error from server (ServiceUnavailable): the server is currently unable to handle the request (get nodes.metrics.k8s.io)
      1. oc get pods -A -o wide | grep -v "Running" | grep -v "Completed"
        NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
        nfs-provisioner nfs-client-provisioner-84766f5b6d-6pscv 0/1 CreateContainerError 0 (4d19h ago) 6d18h 10.128.2.9 worker-1 <none> <none>
        openshift-apiserver apiserver-9fc69858-9tkhg 1/2 CreateContainerError 11 (43h ago) 6d17h 10.128.0.47 master-0 <none> <none>
        openshift-apiserver apiserver-9fc69858-p9x8t 1/2 CreateContainerError 6 (25h ago) 6d18h 10.130.0.32 master-2 <none> <none>
        openshift-authentication-operator authentication-operator-656f485cc6-8rtk8 0/1 CreateContainerError 6 (3d23h ago) 4d8h 10.130.0.62 master-2 <none> <none>
        openshift-authentication oauth-openshift-54bcbd5487-nhkcv 0/1 CreateContainerError 1 (44h ago) 4d8h 10.130.0.70 master-2 <none> <none>
        openshift-cluster-storage-operator csi-snapshot-controller-74cbcb446f-cf572 0/1 CreateContainerError 6 (3d23h ago) 4d8h 10.130.0.60 master-2 <none> <none>
        openshift-cluster-storage-operator csi-snapshot-controller-74cbcb446f-hz6wr 0/1 CreateContainerError 80 (3d23h ago) 6d18h 10.128.0.4 master-0 <none> <none>
        openshift-cluster-version cluster-version-operator-cdd5f5c56-t6d2j 0/1 CreateContainerError 7 (43h ago) 6d18h 9.47.87.104 master-0 <none> <none>
        openshift-config-operator openshift-config-operator-58684b8cc5-7k658 0/1 CreateContainerError 6 (43h ago) 4d8h 10.128.0.69 master-0 <none> <none>
        openshift-console-operator console-operator-7b4ddbcf8d-2h4vw 0/1 CreateContainerError 53 (4d8h ago) 6d18h 10.130.0.22 master-2 <none> <none>
        openshift-console console-649f685c96-g7h96 0/1 CreateContainerError 4 (44h ago) 6d18h 10.130.0.24 master-2 <none> <none>
        openshift-console console-649f685c96-jzgx7 0/1 CreateContainerError 3 (43h ago) 6d18h 10.128.0.39 master-0 <none> <none>
        openshift-image-registry image-pruner-27156960--1-pwn2z 0/1 ContainerCreating 0 4d10h <none> worker-1 <none> <none>
        openshift-kube-apiserver-operator kube-apiserver-operator-54cbc7b859-7jg5w 0/1 CreateContainerError 2 (25h ago) 4d8h 10.130.0.68 master-2 <none> <none>
        openshift-kube-apiserver kube-apiserver-master-0 4/5 CreateContainerError 13 (43h ago) 5d18h 9.47.87.104 master-0 <none> <none>
        openshift-kube-apiserver kube-apiserver-master-2 4/5 CreateContainerError 6 (4d7h ago) 5d18h 9.47.87.106 master-2 <none> <none>
        openshift-kube-controller-manager kube-controller-manager-master-0 3/4 CreateContainerError 88 (3d23h ago) 6d18h 9.47.87.104 master-0 <none> <none>
        openshift-kube-controller-manager kube-controller-manager-master-1 2/4 CreateContainerError 108 (4d ago) 6d18h 9.47.87.110 master-1 <none> <none>
        openshift-kube-controller-manager kube-controller-manager-master-2 3/4 CreateContainerError 74 (4d7h ago) 6d18h 9.47.87.106 master-2 <none> <none>
        openshift-kube-scheduler openshift-kube-scheduler-master-0 2/3 CreateContainerError 67 (2d10h ago) 6d18h 9.47.87.104 master-0 <none> <none>
        openshift-kube-scheduler openshift-kube-scheduler-master-1 2/3 CreateContainerError 59 (4d3h ago) 6d18h 9.47.87.110 master-1 <none> <none>
        openshift-marketplace certified-operators-k7r4p 0/1 ImageInspectError 0 (5d1h ago) 5d2h 10.128.3.108 worker-1 <none> <none>
        openshift-marketplace certified-operators-l7gv2 0/1 ImageInspectError 0 5d1h 10.128.3.126 worker-1 <none> <none>
        openshift-marketplace community-operators-656xf 0/1 ImageInspectError 0 (5d1h ago) 5d1h 10.128.3.118 worker-1 <none> <none>
        openshift-marketplace community-operators-znvc2 0/1 ImageInspectError 0 5d1h 10.128.3.127 worker-1 <none> <none>
        openshift-marketplace marketplace-operator-54c6798b65-77vwn 0/1 CreateContainerError 6 (3d23h ago) 4d8h 10.128.0.62 master-0 <none> <none>
        openshift-marketplace redhat-marketplace-h4rdg 0/1 ContainerCreating 0 5d1h <none> worker-1 <none> <none>
        openshift-marketplace redhat-marketplace-zk6tn 0/1 ImageInspectError 0 (5d1h ago) 6d14h 10.128.2.87 worker-1 <none> <none>
        openshift-marketplace redhat-operators-9k5dj 0/1 ImageInspectError 0 (5d1h ago) 5d1h 10.128.3.116 worker-1 <none> <none>
        openshift-marketplace redhat-operators-9ldp7 0/1 ImageInspectError 0 5d1h 10.128.3.124 worker-1 <none> <none>
        openshift-monitoring alertmanager-main-1 0/5 ContainerCreating 0 4d7h <none> worker-1 <none> <none>
        openshift-monitoring grafana-8657cf6659-dx97s 0/2 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-monitoring kube-state-metrics-7b6988cf78-7x8sw 0/3 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-monitoring openshift-state-metrics-5785f48976-nlzv5 0/3 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-monitoring prometheus-adapter-5f6ff5849b-r79hm 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
        openshift-monitoring prometheus-adapter-6bcc5ddf97-krzsn 0/1 Pending 0 3d23h <none> <none> <none> <none>
        openshift-monitoring prometheus-adapter-6bcc5ddf97-qq7df 0/1 ContainerCreating 0 3d23h <none> worker-1 <none> <none>
        openshift-monitoring prometheus-k8s-0 6/7 CreateContainerError 0 (3d14h ago) 4d7h 10.131.0.249 worker-0 <none> <none>
        openshift-monitoring prometheus-k8s-1 0/7 Init:0/1 0 4d7h <none> worker-1 <none> <none>
        openshift-monitoring telemeter-client-7b954d746b-fspb5 0/3 ContainerCreating 0 4d9h <none> worker-1 <none> <none>
        openshift-network-diagnostics network-check-source-dfbc58d6f-mb2b5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-oauth-apiserver apiserver-6d5b547944-r2njv 0/1 CreateContainerError 3 (45h ago) 4d8h 10.129.0.85 master-1 <none> <none>
        openshift-oauth-apiserver apiserver-6d5b547944-zqnh6 0/1 CreateContainerError 7 (4d7h ago) 6d18h 10.130.0.9 master-2 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156585--1-7gnj8 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156600--1-bfwnf 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156615--1-hvhzf 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156630--1-bm8dh 0/1 ContainerCreating 0 4d16h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156645--1-nqcsj 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156660--1-gckl4 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156675--1-d2hvq 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156690--1-n6pbn 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156705--1-flx77 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156720--1-ml56f 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156735--1-6brkc 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156750--1-9q4k2 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156765--1-kwch8 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156780--1-8vkj5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156795--1-qg8fw 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156810--1-hz9mm 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156825--1-w2wvk 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156840--1-mv2fd 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156855--1-fpgrn 0/1 ContainerCreating 0 4d12h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156870--1-6c7ck 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156885--1-z58sv 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156900--1-q885s 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156915--1-78mhb 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156930--1-898zx 0/1 ContainerCreating 0 4d11h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156945--1-5grj9 0/1 ContainerCreating 0 4d11h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156960--1-q77nv 0/1 ContainerCreating 0 4d10h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156975--1-tqr9j 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27156990--1-87mdj 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157005--1-rcr7m 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157020--1-x6jdv 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157035--1-5mhjc 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157050--1-8nbjz 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157065--1-xfvms 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157080--1-g5f9q 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157095--1-tcch5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157110--1-jh8x5 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157125--1-lfcl4 0/1 ContainerCreating 0 4d8h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157140--1-pfb9r 0/1 ContainerCreating 0 4d7h <none> worker-1 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157185--1-c99ph 0/1 ContainerCreating 0 4d7h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157200--1-vc9kr 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157215--1-7qxjj 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157230--1-bmr9s 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157245--1-8sjsd 0/1 ContainerCreating 0 4d6h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157260--1-vrchv 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157275--1-tdq9c 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157290--1-v782d 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157305--1-5d5zb 0/1 ContainerCreating 0 4d5h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157320--1-jwh24 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157335--1-gltgf 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157350--1-jbmhc 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157365--1-jbmn8 0/1 ContainerCreating 0 4d4h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager collect-profiles-27157665--1-4spth 0/1 ContainerCreating 0 3d23h <none> worker-0 <none> <none>
        openshift-operator-lifecycle-manager olm-operator-75dcc9cb6-8nt5n 0/1 CreateContainerError 4 (43h ago) 4d8h 10.128.0.65 master-0 <none> <none>
        openshift-operator-lifecycle-manager package-server-manager-7b7646568c-gkx7k 0/1 CreateContainerError 6 (4d7h ago) 4d8h 10.130.0.55 master-2 <none> <none>
        openshift-operator-lifecycle-manager packageserver-76694fb586-9pph6 0/1 CreateContainerError 7 (43h ago) 4d8h 10.128.0.68 master-0 <none> <none>

            dgilmore.fedora Dennis Gilmore
            aprabhu2 Alisha Prabhu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: