Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-16129

Image Pruning and Garbage collector unable to reduce the image count

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Critical Critical
    • None
    • 4.11
    • Image Registry
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • No
    • None
    • None
    • Rejected
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Image Pruning and Garbage collector unable to reduce the image count

      Version-Release number of selected component (if applicable):

       

      How reproducible:

       

      Steps to Reproduce:

      Customer created the case for errors seen while pushing the images to OpenShift integrated registry:
      ~~~
      We are seeing an increase in CI/CD job failures in our development clusters.
      The failures are from podman when pushing to the OpenSHift integrated registry
      Example errors
      time="2023-05-05T16:29:24-04:00" level=warning msg="Failed, retrying in 1s ... (2/3). Error: writing blob
      ~~~
      Plus they had hit the ETCD DB 8GB limit and increased the limit to 12GB, and it was updated to them it is not recommended officially. Since it's hardcoded in upstream ETCD, changing it can cause issues in the cluster.
      
      Checked the object count there are 165163 images
      sh-4.4# etcdctl get / --prefix --keys-only | grep -v ^$ | awk -F '/'  '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr
      165163 images
      
      So suggested the garbage collector and image pruning, but these two options are not much help.
       
      https://docs.openshift.com/container-platform/4.11/applications/pruning-objects.html
      
      The CI/CD errors are dropped off and the customers concern is reduce the image count.  

      Actual results:

       

      Expected results:

      Less image count

      Additional info:

       

              fmissi Flavian Missi
              rhn-support-gakendre Gaurav Kendre
              None
              None
              Ge Liu Ge Liu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: