-
Bug
-
Resolution: Obsolete
-
Critical
-
None
-
4.11
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
Important
-
No
-
None
-
None
-
Rejected
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Image Pruning and Garbage collector unable to reduce the image count
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
Customer created the case for errors seen while pushing the images to OpenShift integrated registry:
~~~
We are seeing an increase in CI/CD job failures in our development clusters.
The failures are from podman when pushing to the OpenSHift integrated registry
Example errors
time="2023-05-05T16:29:24-04:00" level=warning msg="Failed, retrying in 1s ... (2/3). Error: writing blob
~~~
Plus they had hit the ETCD DB 8GB limit and increased the limit to 12GB, and it was updated to them it is not recommended officially. Since it's hardcoded in upstream ETCD, changing it can cause issues in the cluster.
Checked the object count there are 165163 images
sh-4.4# etcdctl get / --prefix --keys-only | grep -v ^$ | awk -F '/' '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr
165163 images
So suggested the garbage collector and image pruning, but these two options are not much help.
https://docs.openshift.com/container-platform/4.11/applications/pruning-objects.html
The CI/CD errors are dropped off and the customers concern is reduce the image count.
Actual results:
Expected results:
Less image count
Additional info: