Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-1391

Failed to pull images when Quay configure Swift as the backend storage

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • quay-v3.4.0
    • quay-v3.4.0
    • quay
    • None

      Description of problem:

      Failed to pull images when Quay configure Swift as the backend storage

      Version-Release number of selected component (if applicable):

      registry-proxy.engineering.redhat.com/rh-osbs/quay-quay@sha256:bb58d111dfd3663281f998e10acb49a150245171f0d5215702a2eb75de2f92a9
      registry-proxy.engineering.redhat.com/rh-osbs/quay-quay:v3.4.0-28

      How reproducible:

      Always

      Steps to Reproduce:
      1. Deploy Quay by operator
      2. Configure swift as the backend storage
      3. Push images to Quay

      4. Pull images from Quay

      Actual results:

      failed to pull images

      # skopeo inspect --creds=root:quayroot --tls-verify=false docker://test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle
      FATA[0000] invalid status code from registry 500 (Internal Server Error)
      
      # podman pull --creds=root:quayroot --tls-verify=false test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle
      Trying to pull test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle...
        invalid status code from registry 500 (Internal Server Error)
      Error: error pulling image "test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle": unable to pull test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle: unable to pull image: Error parsing image configuration: invalid status code from registry 500 (Internal Server Error)
      

      Expected results:

      succeed in pulling images

      Additional info:

      logs in quay pod

      nginx stdout | 10.128.2.1 () - - [17/Dec/2020:03:51:06 +0000] "GET /v2/qe/amazon/blobs/sha256:37373184fe69e0fc20370c26317bf4c5b9b843c60b375563d4ee1c7766a89782 HTTP/2.0" 500 141 "-" "Go-http-client/2.0" (0.283 858 0.283)
      nginx stdout | 10.128.2.1 () - - [17/Dec/2020:03:51:06 +0000] "GET /v2/qe/amazon/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/2.0" 499 0 "-" "Go-http-client/2.0" (0.283 74 -)
      securityworker stdout | 2020-12-17 03:51:06,680 [106] [DEBUG] [urllib3.connectionpool] http://test-quay1-clair-app:80 "POST /indexer/api/v1/index_report HTTP/1.1" 201 None
      securityworker stdout | 2020-12-17 03:51:06,690 [106] [DEBUG] [peewee] ('DELETE FROM "manifestsecuritystatus" WHERE ("manifestsecuritystatus"."manifest_id" = %s)', [1])
      securityworker stdout | 2020-12-17 03:51:06,693 [106] [DEBUG] [peewee] ('INSERT INTO "manifestsecuritystatus" ("manifest_id", "repository_id", "index_status", "error_json", "last_indexed", "indexer_hash", "indexer_version", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING "manifestsecuritystatus"."id"', [1, 2, -1, '"failed to fetch layers: encountered error while fetching a layer: fetcher: unexpected status code: 500 500 Internal Server Error"', datetime.datetime(2020, 12, 17, 3, 51, 6, 693361), '955a12d7f80c96ed25a62cc1d21c7cd6', 4, '{}'])
      securityworker stdout | 2020-12-17 03:51:06,701 [106] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
      securityworker stdout | 2020-12-17 03:51:06,705 [106] [DEBUG] [peewee] ('SELECT "t1"."uuid" FROM "imagestorage" AS "t1" INNER JOIN "uploadedblob" AS "t2" ON ("t2"."blob_id" = "t1"."id") WHERE ((("t2"."repository_id" = %s) AND ("t1"."content_checksum" = %s)) AND ("t1"."uploading" = %s)) LIMIT %s OFFSET %s', [3, 'sha256:7535676bcf14d6e467793d6ea0469cb1eab4a8c219d24630f539fadc19492a06', False, 1, 0])
      securityworker stdout | 2020-12-17 03:51:06,710 [106] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id", "t2"."id", "t2"."uuid", "t2"."image_size", "t2"."uncompressed_size", "t2"."uploading", "t2"."cas_path", "t2"."content_checksum" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['9b8280a5-7df0-486f-a252-9ed2f99df66d'])
      gunicorn-registry stdout | 2020-12-17 03:51:06,711 [314] [DEBUG] [urllib3.connectionpool] https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13808 "GET /v1/AUTH_542c6ebd48bf40fa857fc245c7572e30/quay/datastorage/registry/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 200 32
      gunicorn-registry stdout | 2020-12-17 03:51:06,712 [314] [DEBUG] [swiftclient] REQ: curl -i https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13808/v1/AUTH_542c6ebd48bf40fa857fc245c7572e30/quay/datastorage/registry/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -X GET -H "X-Auth-Token: gAAAAABf2tWqdNm4..."
      gunicorn-registry stdout | 2020-12-17 03:51:06,712 [314] [DEBUG] [swiftclient] RESP STATUS: 200 OK
      gunicorn-registry stdout | 2020-12-17 03:51:06,712 [314] [DEBUG] [swiftclient] RESP HEADERS: {'Content-Type': 'application/octet-stream', 'Etag': '54a01009f17bdb7ec1dd1cb427244304', 'Last-Modified': 'Thu, 17 Dec 2020 02:04:25 GMT', 'X-Timestamp': '1608170664.14113', 'Accept-Ranges': 'bytes', 'Content-Length': '32', 'X-Trans-Id': 'txdb30729791044af6981c2-005fdad5aa', 'X-Openstack-Request-Id': 'txdb30729791044af6981c2-005fdad5aa', 'Date': 'Thu, 17 Dec 2020 03:51:06 GMT'}
      securityworker stdout | 2020-12-17 03:51:06,714 [106] [DEBUG] [keystoneclient.auth.identity.v3.base] Making authentication request to https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3/auth/tokens
      gunicorn-registry stdout | 2020-12-17 03:51:06,713 [314] [ERROR] [gunicorn.error] Error handling request /v2/qe/amazon/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
      gunicorn-registry stdout | Traceback (most recent call last):
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
      gunicorn-registry stdout |     self.handle_request(listener_name, req, client, addr)
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
      gunicorn-registry stdout |     super().handle_request(listener_name, req, sock, addr)
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 113, in handle_request
      gunicorn-registry stdout |     for item in respiter:
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/werkzeug/wsgi.py", line 506, in __next__
      gunicorn-registry stdout |     return self._next()
      gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/werkzeug/wrappers/base_response.py", line 45, in _iter_encoded
      gunicorn-registry stdout |     for item in iterable:
      gunicorn-registry stdout |   File "/quay-registry/storage/swift.py", line 267, in stream_read
      gunicorn-registry stdout |     yield data.to_bytes(1, sys.byteorder)
      gunicorn-registry stdout | AttributeError: 'bytes' object has no attribute 'to_bytes'
      gunicorn-registry stdout | 2020-12-17 03:51:06,715 [314] [INFO] [gunicorn.access]  - - [17/Dec/2020:03:51:06 +0000] "GET /v2/qe/amazon/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 500 0 "-" "-"
      securityworker stdout | 2020-12-17 03:51:06,717 [106] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): rhos-d.infra.prod.upshift.rdu2.redhat.com:13000
      securityworker stdout | 2020-12-17 03:51:06,757 [106] [DEBUG] [urllib3.connectionpool] https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000 "POST /v3/auth/tokens HTTP/1.1" 201 10400

              jonathankingfc Jonathan King (Inactive)
              rhn-support-dyan Dongbo Yan
              Dongbo Yan Dongbo Yan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: