Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-6950

Failed to pull images when quay enabled proxy_storage with STS S3 storage

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • quay-v3.11.0
    • quay
    • False
    • None
    • False

      Description of problem:

      Failed to pull images when quay enabled proxy_storage with STS S3 storage

       

      Version-Release number of selected component (if applicable):

      'registry-proxy.engineering.redhat.com/rh-osbs/quay-quay-rhel8@sha256:36de6d97a0c767f355cacec60be0de64fc2306521bc7b92438af159829d010e6',
                                    'registry-proxy.engineering.redhat.com/rh-osbs/quay-quay-rhel8:v3.11.0-23'

       

      How reproducible:

      Always

       

      Steps to Reproduce:
      1. Create STS S3 storage
      2. Setup quay with STS S3 storage and enable proxy_storage, below config.yaml:
        1. DISTRIBUTED_STORAGE_CONFIG:
              local_us:
                  - STSS3Storage
                  - s3_bucket: quay311-sts
                    storage_path: /
                    sts_role_arn: arn:aws:iam::396:role/dyan
                    sts_user_access_key: AKIA
                    sts_user_secret_key: eF9VFb
          DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
              - local_us
          DISTRIBUTED_STORAGE_PREFERENCE:
              - local_us
          FEATURE_PROXY_STORAGE: true

          3. push images and pull images

       

      Actual Results:

      Failed to pull images

      $ podman pull --tls-verify=false --creds=quay:password quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com/qeteam/redhat/ubi9:9.2
      Trying to pull quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com/qeteam/redhat/ubi9:9.2...
      Error: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 502 Bad Gateway 

      quay pod logs:

      gunicorn-registry stdout | 2024-04-01 05:33:09,618 [248] [DEBUG] [endpoints.v2.blob] Returning direct download URL
      gunicorn-registry stdout | 2024-04-01 05:33:09,618 [248] [DEBUG] [app] Ending request: urn:request:03fbbc0d-481d-469f-a770-baf96f105ea1 (/v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779) {'endpoint': 'v2.download_blob', 'request_id': 'urn:request:03fbbc0d-481d-469f-a770-baf96f105ea1', 'remote_addr': '10.128.2.14', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com/v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779', 'path': '/v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779', 'parameters': {}, 'json_body': None, 'confsha': 'bdca5b31', 'user-agent': 'containers/5.26.2 (github.com/containers/image)'}
      gunicorn-registry stdout | 2024-04-01 05:33:09,619 [248] [DEBUG] [data.database] Disconnecting from database.
      gunicorn-registry stdout | 2024-04-01 05:33:09,619 [248] [DEBUG] [peewee.pool] Returning 140669220951360 to pool.
      nginx stdout | 2024/04/01 05:33:09 [error] 96#0: *212 upstream sent too big header while reading response header from upstream, client: 10.128.2.14, server: , request: "GET /v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779", host: "quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com"
      gunicorn-registry stdout | 2024-04-01 05:33:09,619 [248] [INFO] [gunicorn.access] 10.128.2.14 - - [01/Apr/2024:05:33:09 +0000] "GET /v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779 HTTP/1.1" 302 9025 "-" "containers/5.26.2 (github.com/containers/image)"
      gunicorn-web stdout | 2024-04-01 05:33:09,620 [235] [DEBUG] [app] Starting request: urn:request:e39bc54c-5974-445c-a780-3b1bc7a3df24 (/quay-registry/static/502.html) {'X-Forwarded-For': '66.187.232.127, 10.128.2.14'}
      gunicorn-web stdout | 2024-04-01 05:33:09,621 [235] [DEBUG] [app] Ending request: urn:request:e39bc54c-5974-445c-a780-3b1bc7a3df24 (/quay-registry/static/502.html) {'endpoint': None, 'request_id': 'urn:request:e39bc54c-5974-445c-a780-3b1bc7a3df24', 'remote_addr': '10.128.2.14', 'http_method': 'GET', 'original_url': 'https://quayregistry-quay-quay-enterprise-2274.apps.quaytest-2274.qe.devcluster.openshift.com/quay-registry/static/502.html', 'path': '/quay-registry/static/502.html', 'parameters': {}, 'json_body': None, 'confsha': '1b7ca949', 'user-agent': 'containers/5.26.2 (github.com/containers/image)'}
      gunicorn-web stdout | 2024-04-01 05:33:09,622 [235] [INFO] [gunicorn.access] 10.128.2.14 - - [01/Apr/2024:05:33:09 +0000] "GET /quay-registry/static/502.html HTTP/1.0" 308 435 "-" "containers/5.26.2 (github.com/containers/image)"
      nginx stdout | 10.128.2.14 (-) - - [01/Apr/2024:05:33:09 +0000] "GET /v2/qeteam/redhat/ubi9/blobs/sha256:20cef057605e396d5ebea057dcf2bd7702cb3ad13682e7b8e801ed18227cb779 HTTP/1.1" 502 435 "-" "containers/5.26.2 (github.com/containers/image)" (0.261 1767 0.258 : 0.003)

       

      Expected Results:

      Should pull images successfully

       

      Additional Info

      When disable proxy_storage, pulling is success

      FEATURE_PROXY_STORAGE: false

            Unassigned Unassigned
            rhn-support-dyan Dongbo Yan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: