-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
4.13.z, 4.12.z
-
Quality / Stability / Reliability
-
False
-
-
None
-
Moderate
-
No
-
None
-
None
-
Rejected
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Customer is not able to completely remove the image-registry storage. Running the command: "oc exec -n openshift-image-registry $(oc get pod -n openshift-image-registry -l docker-registry=default -o name) -- find /registry/docker/registry/v2/repositories/ -type d -empty -delete" should purge all the empty folders in the image-registry container but some _layers & _manifests folders are not being removed.
Version-Release number of selected component (if applicable):
4.12.14
How reproducible:
Created a buildconfig, and generated multiple builds and created a cronjob based on the image-pruner to run every 15 minutes
Steps to Reproduce:
1.Created a build config and generated multiple builds
# Commands run inside the image-registry container
sh-4.4$ find /registry/docker/registry/v2/ -type d -empty -exec ls -ld {} \;
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 02:03 /registry/docker/registry/v2/repositories/openshift/httpd/_uploads
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 03:41 /registry/docker/registry/v2/repositories/clustercheck/httpd-5/_uploads
sh-4.4$ find /registry/docker/registry/v2/ -type f -exec ls -ld {} \;
-rw-r--r--. 1 1000320000 1000320000 759 Aug 2 01:44 /registry/docker/registry/v2/blobs/sha256/08/0864daa5b477d2d1bfb278da234f6c9367b4db1b27642d0584c3f263830e2ec4/data
-rw-r--r--. 1 1000320000 1000320000 20201 Aug 2 01:44 /registry/docker/registry/v2/blobs/sha256/bc/bc5aeea9e274f03c25013d23c41cca5a85d76a1d60434f459207a34b10bda342/data
-rw-r--r--. 1 1000320000 1000320000 27082 Aug 2 03:06 /registry/docker/registry/v2/blobs/sha256/bc/bc9d64a8aab173b716227d3fdd1f1177aaee1802ac877e9308ca9ef21c191d15/data
-rw-r--r--. 1 1000320000 1000320000 7515417 Aug 2 01:44 /registry/docker/registry/v2/blobs/sha256/e5/e53ac5fae1ac340de75c4c78c6eea9df409b45b2ffee95cd8085a8ed3b9cbf6c/data
-rw-r--r--. 1 1000320000 1000320000 79767891 Aug 2 01:44 /registry/docker/registry/v2/blobs/sha256/d4/d43c95783c3d99a3c275f4c278f8d68a1dfda166c399fd55aee8c1dce7d76611/data
-rw-r--r--. 1 1000320000 1000320000 38894219 Aug 2 01:44 /registry/docker/registry/v2/blobs/sha256/f5/f5cda89dc6728bd37f3d0187a3b0ca20f0c93cae45075fddc9b2ef362e3dc634/data
-rw-r--r--. 1 1000320000 1000320000 760 Aug 2 02:03 /registry/docker/registry/v2/blobs/sha256/c5/c5c6ceaaf5778c041194a492136e8ce20ac2933c25847447e91c19b1e42539f5/data
-rw-r--r--. 1 1000320000 1000320000 37853 Aug 2 03:09 /registry/docker/registry/v2/blobs/sha256/c5/c5c1ac68e51c9c68cff62f7ab7143a39f6ede2395f94eb979e460281152e526c/data
-rw-r--r--. 1 1000320000 1000320000 19280 Aug 2 02:03 /registry/docker/registry/v2/blobs/sha256/0d/0dfa6f95b3a9c13d01e06959022479eec4f331f70f01c762d320de1205003c24/data
[... Omitted Outout ...]
-rw-r--r--. 1 1000320000 1000320000 71 Aug 2 02:52 /registry/docker/registry/v2/repositories/clustercheck/httpd-
2. created a cronjob based on the image-pruner to run every 15 minutes
[quickcluster@upi-0 ~]$ oc get cronjob -n openshift-image-registry image-pruner-x
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
image-pruner-x 0,15,30,45 * * * * False 0 47s 86mThe pruner successfully cleaned up the blobs and links
[
quickcluster@upi-0 ~]$ oc get jobs -n openshift-image-registry
NAME COMPLETIONS DURATION AGE
image-pruner-28180800 1/1 4s 28h
image-pruner-28182240 1/1 4s 4h34m
image-pruner-x-28182480 1/1 4s 34m
image-pruner-x-28182495 1/1 4s 19m
[quickcluster@upi-0 ~]$ oc logs -n openshift-image-registry image-pruner-x-28182480-q48nh
I0802 04:00:01.340730 7 prune.go:348] Creating image pruner with keepYoungerThan=1h0m0s, keepTagRevisions=3, pruneOverSizeLimit=<nil>, allImages=true
Deleting layer link sha256:21d362ca1842a197ebb8795407c62c1da2d52788654cf1dbe939c39dc0a369cb in repository clustercheck/httpd-5
Deleting layer link sha256:2e8407a819d8b1acee12cd5ef1f275b470466c0842900103bee1b85c5f0e5064 in repository clustercheck/httpd-5
Deleting layer link sha256:c2633bb5d8f3bbe07881a0ed8d0570f6ef4327067491ebd1d1f1e885fe6f6c1d in repository clustercheck/httpd-5
Deleting layer link sha256:62dc6b25d5690305f64cf955b7cbd939362dfba83a91e1e70c9ce6b571123ce1 in repository clustercheck/httpd-5
Deleting layer link sha256:93759ea83aefb47578dd6b976c92f7e87cbd7ccde65fa31a2de3204742a0b93c in repository clustercheck/httpd-5
Deleting layer link sha256:4eb708d16d031a6c9426f25a8182b06ad1121b23d43a4de15d0ea795047bfe1c in repository clustercheck/httpd-5
Deleting layer link sha256:bacf724179b830d0e9fcf2f652c8117b1e714c0264d78ef3b37a9852945cee04 in repository clustercheck/httpd-5
Deleting layer link sha256:6af4d82f344ce8203a7f0b4148a6d26a1beb8d962520182ad61a5bc11f77048a in repository clustercheck/httpd-5
Deleting 4 items from image stream clustercheck/httpd-5
Deleting manifest link sha256:b247c8afe88c728e8ff8f9a4904183a5a85dc6d271428b2317bf9e617950e147 in repository clustercheck/httpd-5
Deleting manifest link sha256:87bfb3fb25f5e61d56d22eece56d52f8298e238d0d5ed96b9f5e02de2e55f4f9 in repository clustercheck/httpd-5
Deleting manifest link sha256:73a82023803865f868cc9c02c8c9b44b553e886144d161a249b0bb8fece9f95d in repository clustercheck/httpd-5
[... Omitted Output ...]
Deleting image sha256:87bfb3fb25f5e61d56d22eece56d52f8298e238d0d5ed96b9f5e02de2e55f4f9
Deleting image sha256:b247c8afe88c728e8ff8f9a4904183a5a85dc6d271428b2317bf9e617950e147
Summary: deleted 4 image object(s), deleted 4 image stream tag item(s), updated 1 image stream(s), deleted 8 layer link(s), deleted 4 manifest link(s), deleted 12 blob(s)=> 33 objects deleted (9 internal objects & 24 physical files)
quickcluster@upi-0 ~]$ oc logs -n openshift-image-registry image-pruner-x-28182495-fh268
I0802 04:15:01.330504 7 prune.go:348] Creating image pruner with keepYoungerThan=1h0m0s, keepTagRevisions=3, pruneOverSizeLimit=<nil>, allImages=true
Deleting layer link sha256:b4df694341ea77641d14de770eac8bb41cc11e137ad7029d8365a7f66e3d50d6 in repository clustercheck/httpd-5
Deleting layer link sha256:4753e32fa0d3631d2bf499ca793d090af8d2159e0fc3c4684dd5d89a1b1093ab in repository clustercheck/httpd-5
Deleting layer link sha256:c5c1ac68e51c9c68cff62f7ab7143a39f6ede2395f94eb979e460281152e526c in repository clustercheck/httpd-5
Deleting layer link sha256:989777426f396b88bf80b122518f734d09a32bab2a78b7229b3f3ed2a998e26b in repository clustercheck/httpd-5
Deleting layer link sha256:3832843cc472c90405500c9de210eccbd1521873b831a18807ea07c451357bc7 in repository clustercheck/httpd-5
Deleting layer link sha256:67768a45755ee523e6e1b910ab8188c86b68f7cd9d2e9af92c6c50b77d9abb1d in repository clustercheck/httpd-5
Deleting layer link sha256:cfaaa26b7fd111137e9bbba48580227122b5b71e7a65764ce36f500ad7505e66 in repository clustercheck/httpd-5
Deleting layer link sha256:bc9d64a8aab173b716227d3fdd1f1177aaee1802ac877e9308ca9ef21c191d15 in repository clustercheck/httpd-5
Deleting layer link sha256:a46a48eca44808fd9ebe947ccb756c161d78629b367081be5080e7ea67e5da4e in repository clustercheck/httpd-5
Deleting layer link sha256:25405f38fa93ad671fa406765a1077c1e48bcefe22a95cfc9b97ff05f802f4c2 in repository clustercheck/httpd-5
Deleting layer link sha256:8ee91b645d55060dceb2358c743f4d22674841ed045fa9495dd963f8587e79da in repository clustercheck/httpd-5
Deleting layer link sha256:dfce26346e5c9693b08e43c570243312e749dd62ad7232fe47b73819241163c0 in repository clustercheck/httpd-5
Deleting layer link sha256:65548f78d0d1b50d1683ab593b1ebe29754856415ce32c54c065efc2afdb166b in repository clustercheck/httpd-5
Deleting layer link sha256:a295652dfa61c66922ffd22233ff9b38d581865e0e4e99be5c905110a2b9e704 in repository clustercheck/httpd-5
Deleting 7 items from image stream clustercheck/httpd-5
Deleting manifest link sha256:8e3317f5b797b56bb00f57a986020a7d19346273bfbfa454292532e387e13792 in repository clustercheck/httpd-5
Deleting manifest link sha256:e0c828ea409331efc7d2e28ccc82025957b0cea38cb0eec3cd09049490393a49 in repository clustercheck/httpd-5
[... Omitted Output ...]
Deleting blob sha256:8e3317f5b797b56bb00f57a986020a7d19346273bfbfa454292532e387e13792
Deleting blob sha256:5eedd29334767a4b34c7392ee5a10973ce7ec1dec66c7a305c9fbce6ae5a10a2
Deleting image sha256:8e3317f5b797b56bb00f57a986020a7d19346273bfbfa454292532e387e13792
Deleting image sha256:5eedd29334767a4b34c7392ee5a10973ce7ec1dec66c7a305c9fbce6ae5a10a2
Summary: deleted 7 image object(s), deleted 7 image stream tag item(s), updated 1 image stream(s), deleted 14 layer link(s), deleted 7 manifest link(s), deleted 21 blob(s)=> 57 Objects deleted (15 internal objects & 42 physical files)
24 + 42 = 66 files deleted
Then checking in the image-registry container, there is 49 files left, and many empty folders:
sh-4.4$ find /registry/docker/registry/v2/ -type f -exec ls -ld {} \; | wc -l
49
sh-4.4$ find /registry/docker/registry/v2/ -type d -empty | wc -l
sh-4.4$ find /registry/docker/registry/v2/ -type d -empty -exec ls -ld {} \;
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 04:00 /registry/docker/registry/v2/blobs/sha256/ba
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 04:00 /registry/docker/registry/v2/blobs/sha256/6a
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 04:00 /registry/docker/registry/v2/blobs/sha256/44
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 04:00 /registry/docker/registry/v2/blobs/sha256/93
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 04:00 /registry/docker/registry/v2/blobs/sha256/4e
.....
.....
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 04:15 /registry/docker/registry/v2/repositories/clustercheck/httpd-5/_manifests/revisions/sha256/1d33e2bd941e0542cc5a4265226cf878be6efb8c8a04962db19f43379bb35046
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 04:15 /registry/docker/registry/v2/repositories/clustercheck/httpd-5/_manifests/revisions/sha256/e0c828ea409331efc7d2e28ccc82025957b0cea38cb0eec3cd09049490393a49
drwxr-sr-x. 2 1000320000 1000320000 6 Aug 2 04:15 /registry/docker/registry/v2/repositories/clustercheck/httpd-5/_manifests/revisions/sha256/8e3317f5b797b56bb00f57a986020a7d19346273bfbfa454292532e387e13792
Actual results:
The blobs folders/data have been successfully removed, the remaining folders are simply some placeholders for any blobs SHA256 numbers (so only up to 256 folders). But in another hand, the repositories folders _layers & _manifests are not cleaned up and still contains the tree structures of the image TAG hosting the links.
Expected results:
The repositories folders _layers & _manifests are not cleaned up and still contains the tree structures of the image TAG hosting the links.
Additional info: