-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
quay-v3.12.0, quay-v3.15.1
For an image that is cached locally in Quay, the expectation is that the image is pulled from cache without issues. Currently, it seems that regardless of the state of the image in Quay, proxy functionality calls the upstream registry to test whether it's available or not and only then provides access to the image. This causes issues if the upstream registry is not available for any reason whatsoever. Example:
2025-01-08T14:27:39.206664474Z gunicorn-registry stdout | 2025-01-08 14:27:39,205 [245] [DEBUG] [app] Starting request: urn:request:3c13d11c-cc4e-434b-b577-df71b84cda83 (/v2/bitnami/minio-client/manifests/2024.11.17) {'X-Forwarded-For': '10.1.5.1'}
...
2025-01-08T14:27:39.211566227Z gunicorn-registry stdout | 2025-01-08 14:27:39,211 [245] [DEBUG] [data.cache.impl] Checking cache for key upstream_token__bitnami/docker.io/bitnami/bitnami/minio-client
2025-01-08T14:27:39.212103820Z gunicorn-registry stdout | 2025-01-08 14:27:39,211 [245] [DEBUG] [data.cache.impl] Found no result in cache for key upstream_token__bitnami/docker.io/bitnami/bitnami/minio-client; calling loader
2025-01-08T14:27:39.212103820Z gunicorn-registry stdout | 2025-01-08 14:27:39,211 [245] [DEBUG] [data.cache.impl] Got loaded result for key upstream_token__bitnami/docker.io/bitnami/bitnami/minio-client: None
2025-01-08T14:27:39.212103820Z gunicorn-registry stdout | 2025-01-08 14:27:39,211 [245] [DEBUG] [data.cache.impl] Not caching loaded result for key upstream_token__bitnami/docker.io/bitnami/bitnami/minio-client: None
2025-01-08T14:27:39.213753947Z gunicorn-registry stdout | 2025-01-08 14:27:39,213 [245] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): registry-1.docker.io:443
...
2025-01-08T14:28:39.213202168Z nginx stdout | 2025/01/08 14:28:39 [error] 93#0: *4407 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.1.5.1, server: _, request: "GET /v2/bitnami/minio-client/manifests/2024.11.17 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock/v2/bitnami/minio-client/manifests/2024.11.17", host: "QUAY_HOSTNAME"
2025-01-08T14:28:39.213495250Z nginx stdout | 10.1.5.1 (-) - - [08/Jan/2025:14:28:39 +0000] "GET /v2/bitnami/minio-client/manifests/2024.11.17 HTTP/1.1" 504 167 "-" "containers/5.32.2 (github.com/containers/image)" (60.008 1487 60.008)
In this case, connection to registry-1.docker.io was explicitly disabled but after manifest was already cached in Quay. The result was a 504 Gateway Timeout reported and a failed pull.
