Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-66231

ABI vSphere Installation Failing Due to “No Space Left on Device”

XMLWordPrintable

    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • Rejected
    • None
    • In Progress
    • Bug Fix
    • Hide
      Before this update, changes in RHCOS image layering increased space usage on the rendezvous node’s ephemeral temporary file system to approximately 9.4GB during cluster bootstrapping. Because the ephemeral temporary file system is capped at 50% of available RAM, installation would fail on hosts with less than 19GiB of memory due to insufficient space for container images. With this release, the additional data has been moved to a separate temporary file system. As a result, any rendezvous host meeting the minimum RAM requirement for a control plane node (16GB) now has sufficient capacity to successfully bootstrap a cluster. (link:https://issues.redhat.com/browse/OCPBUGS-66231[OCPBUGS-66231])
      Show
      Before this update, changes in RHCOS image layering increased space usage on the rendezvous node’s ephemeral temporary file system to approximately 9.4GB during cluster bootstrapping. Because the ephemeral temporary file system is capped at 50% of available RAM, installation would fail on hosts with less than 19GiB of memory due to insufficient space for container images. With this release, the additional data has been moved to a separate temporary file system. As a result, any rendezvous host meeting the minimum RAM requirement for a control plane node (16GB) now has sufficient capacity to successfully bootstrap a cluster. (link: https://issues.redhat.com/browse/OCPBUGS-66231 [ OCPBUGS-66231 ])
    • None
    • None
    • None
    • None

      This is a clone of issue OCPBUGS-62790. The following is the description of the original issue:
      —
      Description of problem:

      vSphere installation is failing during the image unpacking with a “no space left on device” error.This issue is consistently observed in Compact and HA installations, while it occurs intermittently in SNO (Single Node OpenShift) installations.  

      Version-Release number of selected component (if applicable):

      4.19, 4.20, 4.21

      How reproducible:

      Always

      Steps to Reproduce:

          1. Create agent ISO
    2. Boot the created ISO on vSphere vm's

      Actual results:

      MESSAGE=E1006 22:28:58.992836    6040 pod_workers.go:1301] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"kube-controller-manager\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4314d3f33ac9e16f8dfab5cdf95844747661d550d4e36546b8716ea5896e4ddc\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: writing blob: storing blob to file \\\"/var/tmp/container_images_storage979484291/1\\\": write /var/tmp/container_images_storage979484291/1: no space left on device; artifact err: provided artifact is a container image\", failed to \"StartContainer\" for \"cluster-policy-controller\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cluster-policy-controller pod=bootstrap-kube-controller-manager-ci-op-7qd07q7l-a131e-master-0_kube-system(37efc67fc88c33fd24f134e445557f49)\"]" pod="kube-system/bootstrap-kube-controller-manager-ci-op-7qd07q7l-a131e-master-0" podUID="37efc67fc88c33fd24f134e445557f49"

      Expected results:

      The installation should be successful.

      Agent Gather: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-eng-agent-qe-infra-release-4.20-amd64-nightly-vsphere-agent-ha-day2-add-nodes-f7/1975319672735666176/artifacts/vsphere-agent-ha-day2-add-nodes-f7/cucushift-agent-gather/artifacts/agent-gather.tar.xz 

      VM Configuration: 

      Memory=16384 
CPU=4
Disk Size=120GB

              zabitter Zane Bitter
              rhn-support-mhans Manoj Hans
              zhenying niu
              None
              zhenying niu zhenying niu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: