Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-76450

Massive MCO container causing memory problems during bootstrapping

    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • Yes
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The PR for MCO-1117 optimised local developer builds of the MCO container by mounting a cache directory shared between the build stages and removing the commands to clear the cache.

      Due to RHEL-129120, the directory mounts had to be removed by the fix for OCPBUGS-65726, which was also backported all the way to 4.17. However, the original patch was not reverted in its entirety and this change did not reinstate the code to clear the cache. Therefore we are now shipping a 1.2GiB yum cache inside the MCO container, tripling the size of the image.

      This appears to be causing all kinds of problems during bootstrapping in environments where disk space is constrained - particularly assisted installs (including ABI, ZTP) where the container must be pulled from within a running live ISO. In particular the ABI HA job is permafailing because workers do not have enough memory to pull the MCO image while running from the ephemeral ramdisk (which is limited to half the size of the available RAM). It may also be related to the failure of https://github.com/openshift-metal3/dev-scripts/pull/1819 to pass CI, and to various memory-related CI issues reported in assisted e2e jobs.

              zzlotnik@redhat.com Zack Zlotnik
              zabitter Zane Bitter
              None
              None
              Sergio Regidor de la Rosa Sergio Regidor de la Rosa
              None
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: