-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
quay-v3.7.0
I created a namespace called redhat and proceeded to proxy the ubi8/ubi:latest image. Everything went fine. The manifest list was created and the image was successfully proxied. However, the new image is now stuck in the queued status because of missing layers:
securityworker stdout | 2022-05-30 12:11:39,061 [97] [DEBUG] [data.secscan_model.secscan_v4_model] Indexing manifest [4] 3/ubi8/ubi@sha256:88b67c5c3d7bc900e0dc77c058601c618758e3c79d468ebfe446e91c45657b46
The manifest creates the following errors in Clair:
{"level":"warn","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:88b67c5c3d7bc900e0dc77c058601c618758e3c79d468ebfe446e91c45657b46","state":"FetchLayers","error":"encountered error while fetching a layer: fetcher: unexpected status code: 404 Not Found (body starts: \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>datastorage/registry/sha256/54/545277d800059b32cf03377a9301094e9ac8aa4bb42d809766d7355ca9aa8652</Key><BucketName>quay</BucketName><\")","time":"2022-05-30T12:12:39Z","message":"layers fetch failure"}
{"level":"info","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:88b67c5c3d7bc900e0dc77c058601c618758e3c79d468ebfe446e91c45657b46","state":"FetchLayers","time":"2022-05-30T12:12:39Z","message":"layers fetch done"}
{"level":"error","component":"internal/indexer/controller/Controller.Index","manifest":"sha256:88b67c5c3d7bc900e0dc77c058601c618758e3c79d468ebfe446e91c45657b46","state":"FetchLayers","error":"failed to fetch layers: encountered error while fetching a layer: fetcher: unexpected status code: 404 Not Found (body starts: \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>datastorage/registry/sha256/54/545277d800059b32cf03377a9301094e9ac8aa4bb42d809766d7355ca9aa8652</Key><BucketName>quay</BucketName><\")","time":"2022-05-30T12:12:39Z","message":"error during scan"}
{"level":"info","component":"libindex/Libindex.Index","manifest":"sha256:88b67c5c3d7bc900e0dc77c058601c618758e3c79d468ebfe446e91c45657b46","time":"2022-05-30T12:12:39Z","message":"index request done"}
{"level":"info","component":"httptransport/New","remote_addr":"172.17.0.1:38036","method":"POST","request_uri":"/indexer/api/v1/index_report","status":500,"duration":55.416127,"time":"2022-05-30T12:12:39Z","message":"handled HTTP request"}
When inspecting the manifest, it should contain the following layers:
# skopeo inspect --raw --tls-verify=false docker://quay.tardis/redhat/ubi8/ubi@sha256:88b67c5c3d7bc900e0dc77c058601c618758e3c79d468ebfe446e91c45657b46 { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 4366, "digest": "sha256:1264065f6ae851d6a33d7be03ffde100356592e385b9b72f65f91b5d9b944b92" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 78433080, "digest": "sha256:f70d60810c69edad990aaf0977a87c6d2bcc9cd52904fa6825f08507a9b6e7bc" }, { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 1795, "digest": "sha256:545277d800059b32cf03377a9301094e9ac8aa4bb42d809766d7355ca9aa8652" } ] }
However, when inspecting the bucket directly, layer f7 does not exist:
root@dalek:~# s3cmd ls s3://quay/datastorage/registry/sha256/ | grep f7 root@dalek:~#
The image is definitely downloaded correctly locally:
# docker images | grep ubi
quay.tardis/redhat/ubi8/ubi latest 1264065f6ae8 3 weeks ago 207MB
Then I tried pulling with podman and sure enough the layer location and the layer itself were created successfully in storage:
# s3cmd ls s3://quay/datastorage/registry/sha256/f7/
2022-05-30 12:18 78433080 s3://quay/datastorage/registry/sha256/f7/f70d60810c69edad990aaf0977a87c6d2bcc9cd52904fa6825f08507a9b6e7bc
I also created a proxy docker namespace and tried to proxy an image from there and that image
{mysql}was proxied correctly, all layers were saved. I don't see anything obviously wrong in Quay's logs and I don't understand why would that layer be missing from storage even though in both cases pull succeeded.
Full Quay logs uploaded.
