Critical Description of problem:
Failed to pull images when quay enabled proxy_storage with STS S3 storage
Version-Release number of selected component (if applicable):
'registry-proxy.engineering.redhat.com/rh-osbs/quay-quay-rhel8@sha256:36de6d97a0c767f355cacec60be0de64fc2306521bc7b92438af159829d010e6',
'registry-proxy.engineering.redhat.com/rh-osbs/quay-quay-rhel8:v3.11.0-23'
How reproducible:
Always
Steps to Reproduce:
- Create STS S3 storage
- Setup quay with STS S3 storage and enable proxy_storage, below config.yaml:
DISTRIBUTED_STORAGE_CONFIG: local_us: - STSS3Storage - s3_bucket: quay311-sts storage_path: / sts_role_arn: arn:aws:iam::396:role/dyan sts_user_access_key: AKIA sts_user_secret_key: eF9VFb DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: - local_us DISTRIBUTED_STORAGE_PREFERENCE: - local_us FEATURE_PROXY_STORAGE: true
3. push images and pull images
Actual Results:
Failed to pull images
$ podman pull --tls-verify=false --creds=quay:password quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com/qeteam/redhat/ubi9:9.2
Trying to pull quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com/qeteam/redhat/ubi9:9.2...
Error: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 502 Bad Gateway
quay pod logs:
gunicorn-registry stdout | 2024-04-01 05:33:09,618 [248] [DEBUG] [endpoints.v2.blob] Returning direct download URL
gunicorn-registry stdout | 2024-04-01 05:33:09,618 [248] [DEBUG] [app] Ending request: urn:request:03fbbc0d-481d-469f-a770-baf96f105ea1 (/v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779) {'endpoint': 'v2.download_blob', 'request_id': 'urn:request:03fbbc0d-481d-469f-a770-baf96f105ea1', 'remote_addr': '10.128.2.14', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com/v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779', 'path': '/v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779', 'parameters': {}, 'json_body': None, 'confsha': 'bdca5b31', 'user-agent': 'containers/5.26.2 (github.com/containers/image)'}
gunicorn-registry stdout | 2024-04-01 05:33:09,619 [248] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-01 05:33:09,619 [248] [DEBUG] [peewee.pool] Returning 140669220951360 to pool.
nginx stdout | 2024/04/01 05:33:09 [error] 96#0: *212 upstream sent too big header while reading response header from upstream, client: 10.128.2.14, server: , request: "GET /v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779", host: "quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com"
gunicorn-registry stdout | 2024-04-01 05:33:09,619 [248] [INFO] [gunicorn.access] 10.128.2.14 - - [01/Apr/2024:05:33:09 +0000] "GET /v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779 HTTP/1.1" 302 9025 "-" "containers/5.26.2 (github.com/containers/image)"
gunicorn-web stdout | 2024-04-01 05:33:09,620 [235] [DEBUG] [app] Starting request: urn:request:e39bc54c-5974-445c-a780-3b1bc7a3df24 (/quay-registry/static/502.html) {'X-Forwarded-For': '66.187.232.127, 10.128.2.14'}
gunicorn-web stdout | 2024-04-01 05:33:09,621 [235] [DEBUG] [app] Ending request: urn:request:e39bc54c-5974-445c-a780-3b1bc7a3df24 (/quay-registry/static/502.html) {'endpoint': None, 'request_id': 'urn:request:e39bc54c-5974-445c-a780-3b1bc7a3df24', 'remote_addr': '10.128.2.14', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com/quay-registry/static/502.html', 'path': '/quay-registry/static/502.html', 'parameters': {}, 'json_body': None, 'confsha': '1b7ca949', 'user-agent': 'containers/5.26.2 (github.com/containers/image)'}
gunicorn-web stdout | 2024-04-01 05:33:09,622 [235] [INFO] [gunicorn.access] 10.128.2.14 - - [01/Apr/2024:05:33:09 +0000] "GET /quay-registry/static/502.html HTTP/1.0" 308 435 "-" "containers/5.26.2 (github.com/containers/image)"
nginx stdout | 10.128.2.14 (-) - - [01/Apr/2024:05:33:09 +0000] "GET /v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779 HTTP/1.1" 502 435 "-" "containers/5.26.2 (github.com/containers/image)" (0.261 1767 0.258 : 0.003)
Expected Results:
Should pull images successfully
Additional Info
When disable proxy_storage, pulling is success
FEATURE_PROXY_STORAGE: false
