-
Bug
-
Resolution: Done
-
Blocker
-
quay-v3.5.0
-
False
-
False
-
Undefined
-
Description:
This is an issue found when push image to quay, the result is hit 500 error, checked quay app pod, get error "AttributeError: 'tuple' object has no attribute 'decode'"
Note: this issue can be reproduced when quay use 'AWS S3', 'OCS', 'GCP Object storage' as backend registry storage.
docker push quay350aws3-quay-quay350.apps.quay-perf-549.perfscale.devcluster.openshift.com/qateam/demo2
Using default tag: latest
The push refers to repository [quay350aws3-quay-quay350.apps.quay-perf-549.perfscale.devcluster.openshift.com/qateam/demo2]
e84e86a2ce7e: Retrying in 1 second
received unexpected HTTP status: 500 Internal Server Error
Quay Version:
oc get pod NAME READY STATUS RESTARTS AGE quay350aws3-clair-app-775d5b4557-9bl52 1/1 Running 0 21m quay350aws3-clair-postgres-6f4fb9d849-k2rl2 1/1 Running 1 25m quay350aws3-quay-app-7ccd8d5d6c-fch6q 1/1 Running 0 21m quay350aws3-quay-config-editor-7549d66d5b-dgxbp 1/1 Running 0 21m quay350aws3-quay-database-559944bd5c-mwf2j 1/1 Running 0 21m quay350aws3-quay-mirror-7b6866cfb8-wxb6v 1/1 Running 0 21m quay350aws3-quay-postgres-init-x5lhd 0/1 Completed 0 21m quay350aws3-quay-redis-66cfc6dc8c-v6fg9 1/1 Running 0 25m oc get pod quay350aws3-quay-app-7ccd8d5d6c-fch6q -o json | jq '.spec.containers[0].image' "registry.redhat.io/quay/quay-rhel8@sha256:6f979d2983f16a08908b6090b6d6f7724dc280c80cf7aff9a9c829da104cf46c"
Index image:
Index image v4.7: registry-proxy.engineering.redhat.com/rh-osbs/iib:59430
Quay App POD logs:
gunicorn-registry stdout | 2021-03-25 07:24:31,860 [258] [ERROR] [gunicorn.error] Error handling request /v2/qateam/demo2/blobs/uploads/ gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle gunicorn-registry stdout | self.handle_request(listener_name, req, client, addr) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request gunicorn-registry stdout | super().handle_request(listener_name, req, sock, addr) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request gunicorn-registry stdout | respiter = self.wsgi(environ, resp.start_response) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2463, in __call__ gunicorn-registry stdout | return self.wsgi_app(environ, start_response) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__ gunicorn-registry stdout | return self.app(environ, start_response) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app gunicorn-registry stdout | response = self.handle_exception(e) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1866, in handle_exception gunicorn-registry stdout | reraise(exc_type, exc_value, tb) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise gunicorn-registry stdout | raise value gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app gunicorn-registry stdout | response = self.full_dispatch_request() gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request gunicorn-registry stdout | rv = self.handle_user_exception(e) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception gunicorn-registry stdout | rv = self.handle_user_exception(e) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception gunicorn-registry stdout | reraise(exc_type, exc_value, tb) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise gunicorn-registry stdout | raise value gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request gunicorn-registry stdout | rv = self.dispatch_request() gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request gunicorn-registry stdout | return self.view_functions[rule.endpoint](**req.view_args) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 57, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/auth/registry_jwt_auth.py", line 177, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/__init__.py", line 126, in wrapped gunicorn-registry stdout | return func(namespace_name, repo_name, *args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 120, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 145, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/blob.py", line 238, in start_blob_upload gunicorn-registry stdout | blob_uploader = create_blob_upload(repository_ref, storage, _upload_settings()) gunicorn-registry stdout | File "/quay-registry/data/registry_model/blobuploader.py", line 78, in create_blob_upload gunicorn-registry stdout | new_upload_uuid, upload_metadata = storage.initiate_chunked_upload(location_name) gunicorn-registry stdout | File "/quay-registry/storage/distributedstorage.py", line 27, in wrapper gunicorn-registry stdout | return storage_func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/storage/cloud.py", line 513, in initiate_chunked_upload gunicorn-registry stdout | self._initialize_cloud_conn() gunicorn-registry stdout | File "/quay-registry/storage/cloud.py", line 144, in _initialize_cloud_conn gunicorn-registry stdout | self._cloud_conn = self._session.client("s3", **self._connect_kwargs) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258, in client gunicorn-registry stdout | return self._session.create_client( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 834, in create_client gunicorn-registry stdout | client = client_creator.create_client( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 85, in create_client gunicorn-registry stdout | client_args = self._get_client_args( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 326, in _get_client_args gunicorn-registry stdout | return args_creator.get_client_args( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 99, in get_client_args gunicorn-registry stdout | endpoint = endpoint_creator.create_endpoint( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 285, in create_endpoint gunicorn-registry stdout | if not is_valid_endpoint_url(endpoint_url): gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 993, in is_valid_endpoint_url gunicorn-registry stdout | parts = urlsplit(endpoint_url) gunicorn-registry stdout | File "/usr/lib64/python3.8/urllib/parse.py", line 423, in urlsplit gunicorn-registry stdout | return storage_func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/storage/cloud.py", line 513, in initiate_chunked_upload gunicorn-registry stdout | self._initialize_cloud_conn() gunicorn-registry stdout | File "/quay-registry/storage/cloud.py", line 144, in _initialize_cloud_conn gunicorn-registry stdout | self._cloud_conn = self._session.client("s3", **self._connect_kwargs) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258, in client gunicorn-registry stdout | return self._session.create_client( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 834, in create_client gunicorn-registry stdout | client = client_creator.create_client( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 85, in create_client gunicorn-registry stdout | client_args = self._get_client_args( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 326, in _get_client_args gunicorn-registry stdout | return args_creator.get_client_args( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 99, in get_client_args gunicorn-registry stdout | endpoint = endpoint_creator.create_endpoint( gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 285, in create_endpoint gunicorn-registry stdout | if not is_valid_endpoint_url(endpoint_url): gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 993, in is_valid_endpoint_url gunicorn-registry stdout | parts = urlsplit(endpoint_url) gunicorn-registry stdout | File "/usr/lib64/python3.8/urllib/parse.py", line 423, in urlsplit gunicorn-registry stdout | url, scheme, _coerce_result = _coerce_args(url, scheme) gunicorn-registry stdout | File "/usr/lib64/python3.8/urllib/parse.py", line 124, in _coerce_args gunicorn-registry stdout | return _decode_args(args) + (_encode_result,) gunicorn-registry stdout | File "/usr/lib64/python3.8/urllib/parse.py", line 108, in _decode_args gunicorn-registry stdout | return tuple(x.decode(encoding, errors) if x else '' for x in args) gunicorn-registry stdout | File "/usr/lib64/python3.8/urllib/parse.py", line 108, in <genexpr> gunicorn-registry stdout | return tuple(x.decode(encoding, errors) if x else '' for x in args) gunicorn-registry stdout | AttributeError: 'tuple' object has no attribute 'decode' gunicorn-registry stdout | 2021-03-25 07:24:31,861 [258] [INFO] [gunicorn.access] - - [25/Mar/2021:07:24:31 +0000] "POST /v2/qateam/demo2/blobs/uploads/ HTTP/1.1" 500 0 "-" "-" nginx stdout | 10.131.3.40 () - - [25/Mar/2021:07:24:31 +0000] "POST /v2/qateam/demo2/blobs/uploads/ HTTP/1.1" 500 141 "-" "docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \x5C(darwin\x5C))" (0.011 1455 0.012)
