-
Bug
-
Resolution: Done
-
Blocker
-
quay-v3.4.0
-
None
-
False
-
False
-
Quay Enterprise
-
Undefined
-
Description of problem:
Failed to pull images when Quay configure Swift as the backend storage
Version-Release number of selected component (if applicable):
registry-proxy.engineering.redhat.com/rh-osbs/quay-quay@sha256:bb58d111dfd3663281f998e10acb49a150245171f0d5215702a2eb75de2f92a9
registry-proxy.engineering.redhat.com/rh-osbs/quay-quay:v3.4.0-28
How reproducible:
Always
Steps to Reproduce:
1. Deploy Quay by operator
2. Configure swift as the backend storage
3. Push images to Quay
4. Pull images from Quay
Actual results:
failed to pull images
# skopeo inspect --creds=root:quayroot --tls-verify=false docker://test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle FATA[0000] invalid status code from registry 500 (Internal Server Error) # podman pull --creds=root:quayroot --tls-verify=false test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle Trying to pull test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle... invalid status code from registry 500 (Internal Server Error) Error: error pulling image "test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle": unable to pull test-quay1-quay-quay.apps.preserve-dyan.1215-5z5.qe.rhcloud.com/qe/oracle: unable to pull image: Error parsing image configuration: invalid status code from registry 500 (Internal Server Error)
Expected results:
succeed in pulling images
Additional info:
logs in quay pod
nginx stdout | 10.128.2.1 () - - [17/Dec/2020:03:51:06 +0000] "GET /v2/qe/amazon/blobs/sha256:37373184fe69e0fc20370c26317bf4c5b9b843c60b375563d4ee1c7766a89782 HTTP/2.0" 500 141 "-" "Go-http-client/2.0" (0.283 858 0.283)
nginx stdout | 10.128.2.1 () - - [17/Dec/2020:03:51:06 +0000] "GET /v2/qe/amazon/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/2.0" 499 0 "-" "Go-http-client/2.0" (0.283 74 -)
securityworker stdout | 2020-12-17 03:51:06,680 [106] [DEBUG] [urllib3.connectionpool] http://test-quay1-clair-app:80 "POST /indexer/api/v1/index_report HTTP/1.1" 201 None
securityworker stdout | 2020-12-17 03:51:06,690 [106] [DEBUG] [peewee] ('DELETE FROM "manifestsecuritystatus" WHERE ("manifestsecuritystatus"."manifest_id" = %s)', [1])
securityworker stdout | 2020-12-17 03:51:06,693 [106] [DEBUG] [peewee] ('INSERT INTO "manifestsecuritystatus" ("manifest_id", "repository_id", "index_status", "error_json", "last_indexed", "indexer_hash", "indexer_version", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING "manifestsecuritystatus"."id"', [1, 2, -1, '"failed to fetch layers: encountered error while fetching a layer: fetcher: unexpected status code: 500 500 Internal Server Error"', datetime.datetime(2020, 12, 17, 3, 51, 6, 693361), '955a12d7f80c96ed25a62cc1d21c7cd6', 4, '{}'])
securityworker stdout | 2020-12-17 03:51:06,701 [106] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
securityworker stdout | 2020-12-17 03:51:06,705 [106] [DEBUG] [peewee] ('SELECT "t1"."uuid" FROM "imagestorage" AS "t1" INNER JOIN "uploadedblob" AS "t2" ON ("t2"."blob_id" = "t1"."id") WHERE ((("t2"."repository_id" = %s) AND ("t1"."content_checksum" = %s)) AND ("t1"."uploading" = %s)) LIMIT %s OFFSET %s', [3, 'sha256:7535676bcf14d6e467793d6ea0469cb1eab4a8c219d24630f539fadc19492a06', False, 1, 0])
securityworker stdout | 2020-12-17 03:51:06,710 [106] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id", "t2"."id", "t2"."uuid", "t2"."image_size", "t2"."uncompressed_size", "t2"."uploading", "t2"."cas_path", "t2"."content_checksum" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['9b8280a5-7df0-486f-a252-9ed2f99df66d'])
gunicorn-registry stdout | 2020-12-17 03:51:06,711 [314] [DEBUG] [urllib3.connectionpool] https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13808 "GET /v1/AUTH_542c6ebd48bf40fa857fc245c7572e30/quay/datastorage/registry/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 200 32
gunicorn-registry stdout | 2020-12-17 03:51:06,712 [314] [DEBUG] [swiftclient] REQ: curl -i https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13808/v1/AUTH_542c6ebd48bf40fa857fc245c7572e30/quay/datastorage/registry/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -X GET -H "X-Auth-Token: gAAAAABf2tWqdNm4..."
gunicorn-registry stdout | 2020-12-17 03:51:06,712 [314] [DEBUG] [swiftclient] RESP STATUS: 200 OK
gunicorn-registry stdout | 2020-12-17 03:51:06,712 [314] [DEBUG] [swiftclient] RESP HEADERS: {'Content-Type': 'application/octet-stream', 'Etag': '54a01009f17bdb7ec1dd1cb427244304', 'Last-Modified': 'Thu, 17 Dec 2020 02:04:25 GMT', 'X-Timestamp': '1608170664.14113', 'Accept-Ranges': 'bytes', 'Content-Length': '32', 'X-Trans-Id': 'txdb30729791044af6981c2-005fdad5aa', 'X-Openstack-Request-Id': 'txdb30729791044af6981c2-005fdad5aa', 'Date': 'Thu, 17 Dec 2020 03:51:06 GMT'}
securityworker stdout | 2020-12-17 03:51:06,714 [106] [DEBUG] [keystoneclient.auth.identity.v3.base] Making authentication request to https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3/auth/tokens
gunicorn-registry stdout | 2020-12-17 03:51:06,713 [314] [ERROR] [gunicorn.error] Error handling request /v2/qe/amazon/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
gunicorn-registry stdout | self.handle_request(listener_name, req, client, addr)
gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
gunicorn-registry stdout | super().handle_request(listener_name, req, sock, addr)
gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 113, in handle_request
gunicorn-registry stdout | for item in respiter:
gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/werkzeug/wsgi.py", line 506, in __next__
gunicorn-registry stdout | return self._next()
gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/werkzeug/wrappers/base_response.py", line 45, in _iter_encoded
gunicorn-registry stdout | for item in iterable:
gunicorn-registry stdout | File "/quay-registry/storage/swift.py", line 267, in stream_read
gunicorn-registry stdout | yield data.to_bytes(1, sys.byteorder)
gunicorn-registry stdout | AttributeError: 'bytes' object has no attribute 'to_bytes'
gunicorn-registry stdout | 2020-12-17 03:51:06,715 [314] [INFO] [gunicorn.access] - - [17/Dec/2020:03:51:06 +0000] "GET /v2/qe/amazon/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 500 0 "-" "-"
securityworker stdout | 2020-12-17 03:51:06,717 [106] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): rhos-d.infra.prod.upshift.rdu2.redhat.com:13000
securityworker stdout | 2020-12-17 03:51:06,757 [106] [DEBUG] [urllib3.connectionpool] https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000 "POST /v3/auth/tokens HTTP/1.1" 201 10400