Loading...

Details

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: quay-v3.5.0
Affects Version/s: quay-v3.5.0
Component/s: quay
Labels:
- quay-enterprise

Blocked:
False
Ready:
False
Release Note Text:
Undefined
Git Pull Request:
https://github.com/quay/quay/pull/709

RICE Score:
0

SFDC Cases Links:
SFDC Cases Counter:

Description

Description:

This is an issue found when push image to quay, the result is hit 500 error, checked quay app pod, get error "AttributeError: 'tuple' object has no attribute 'decode'"

Note: this issue can be reproduced when quay use 'AWS S3', 'OCS', 'GCP Object storage' as backend registry storage.

docker push quay350aws3-quay-quay350.apps.quay-perf-549.perfscale.devcluster.openshift.com/qateam/demo2
Using default tag: latest
The push refers to repository [quay350aws3-quay-quay350.apps.quay-perf-549.perfscale.devcluster.openshift.com/qateam/demo2]
e84e86a2ce7e: Retrying in 1 second 
received unexpected HTTP status: 500 Internal Server Error

Quay Version:

oc get pod
NAME                                              READY   STATUS      RESTARTS   AGE
quay350aws3-clair-app-775d5b4557-9bl52            1/1     Running     0          21m
quay350aws3-clair-postgres-6f4fb9d849-k2rl2       1/1     Running     1          25m
quay350aws3-quay-app-7ccd8d5d6c-fch6q             1/1     Running     0          21m
quay350aws3-quay-config-editor-7549d66d5b-dgxbp   1/1     Running     0          21m
quay350aws3-quay-database-559944bd5c-mwf2j        1/1     Running     0          21m
quay350aws3-quay-mirror-7b6866cfb8-wxb6v          1/1     Running     0          21m
quay350aws3-quay-postgres-init-x5lhd              0/1     Completed   0          21m
quay350aws3-quay-redis-66cfc6dc8c-v6fg9           1/1     Running     0          25m

oc get pod quay350aws3-quay-app-7ccd8d5d6c-fch6q -o json | jq '.spec.containers[0].image'
"registry.redhat.io/quay/quay-rhel8@sha256:6f979d2983f16a08908b6090b6d6f7724dc280c80cf7aff9a9c829da104cf46c"

Index image:

Index image v4.7: registry-proxy.engineering.redhat.com/rh-osbs/iib:59430

Quay App POD logs:

gunicorn-registry stdout | 2021-03-25 07:24:31,860 [258] [ERROR] [gunicorn.error] Error handling request /v2/qateam/demo2/blobs/uploads/
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
gunicorn-registry stdout |     self.handle_request(listener_name, req, client, addr)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
gunicorn-registry stdout |     super().handle_request(listener_name, req, sock, addr)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request
gunicorn-registry stdout |     respiter = self.wsgi(environ, resp.start_response)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2463, in __call__
gunicorn-registry stdout |     return self.wsgi_app(environ, start_response)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
gunicorn-registry stdout |     return self.app(environ, start_response)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app
gunicorn-registry stdout |     response = self.handle_exception(e)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1866, in handle_exception
gunicorn-registry stdout |     reraise(exc_type, exc_value, tb)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
gunicorn-registry stdout |     raise value
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
gunicorn-registry stdout |     response = self.full_dispatch_request()
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request
gunicorn-registry stdout |     rv = self.handle_user_exception(e)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
gunicorn-registry stdout |     rv = self.handle_user_exception(e)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
gunicorn-registry stdout |     reraise(exc_type, exc_value, tb)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
gunicorn-registry stdout |     raise value
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
gunicorn-registry stdout |     rv = self.dispatch_request()
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
gunicorn-registry stdout |     return self.view_functions[rule.endpoint](**req.view_args)
gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 57, in wrapper
gunicorn-registry stdout |     return func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/auth/registry_jwt_auth.py", line 177, in wrapper
gunicorn-registry stdout |     return func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/__init__.py", line 126, in wrapped
gunicorn-registry stdout |     return func(namespace_name, repo_name, *args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 120, in wrapper
gunicorn-registry stdout |     return func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 145, in wrapper
gunicorn-registry stdout |     return func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/blob.py", line 238, in start_blob_upload
gunicorn-registry stdout |     blob_uploader = create_blob_upload(repository_ref, storage, _upload_settings())
gunicorn-registry stdout |   File "/quay-registry/data/registry_model/blobuploader.py", line 78, in create_blob_upload
gunicorn-registry stdout |     new_upload_uuid, upload_metadata = storage.initiate_chunked_upload(location_name)
gunicorn-registry stdout |   File "/quay-registry/storage/distributedstorage.py", line 27, in wrapper
gunicorn-registry stdout |     return storage_func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/storage/cloud.py", line 513, in initiate_chunked_upload
gunicorn-registry stdout |     self._initialize_cloud_conn()
gunicorn-registry stdout |   File "/quay-registry/storage/cloud.py", line 144, in _initialize_cloud_conn
gunicorn-registry stdout |     self._cloud_conn = self._session.client("s3", **self._connect_kwargs)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258, in client
gunicorn-registry stdout |     return self._session.create_client(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 834, in create_client
gunicorn-registry stdout |     client = client_creator.create_client(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 85, in create_client
gunicorn-registry stdout |     client_args = self._get_client_args(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 326, in _get_client_args
gunicorn-registry stdout |     return args_creator.get_client_args(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 99, in get_client_args
gunicorn-registry stdout |     endpoint = endpoint_creator.create_endpoint(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 285, in create_endpoint
gunicorn-registry stdout |     if not is_valid_endpoint_url(endpoint_url):
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 993, in is_valid_endpoint_url
gunicorn-registry stdout |     parts = urlsplit(endpoint_url)
gunicorn-registry stdout |   File "/usr/lib64/python3.8/urllib/parse.py", line 423, in urlsplit
gunicorn-registry stdout |     return storage_func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/storage/cloud.py", line 513, in initiate_chunked_upload
gunicorn-registry stdout |     self._initialize_cloud_conn()
gunicorn-registry stdout |   File "/quay-registry/storage/cloud.py", line 144, in _initialize_cloud_conn
gunicorn-registry stdout |     self._cloud_conn = self._session.client("s3", **self._connect_kwargs)
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/boto3/session.py", line 258, in client
gunicorn-registry stdout |     return self._session.create_client(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/session.py", line 834, in create_client
gunicorn-registry stdout |     client = client_creator.create_client(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 85, in create_client
gunicorn-registry stdout |     client_args = self._get_client_args(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 326, in _get_client_args
gunicorn-registry stdout |     return args_creator.get_client_args(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/args.py", line 99, in get_client_args
gunicorn-registry stdout |     endpoint = endpoint_creator.create_endpoint(
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 285, in create_endpoint
gunicorn-registry stdout |     if not is_valid_endpoint_url(endpoint_url):
gunicorn-registry stdout |   File "/usr/local/lib/python3.8/site-packages/botocore/utils.py", line 993, in is_valid_endpoint_url
gunicorn-registry stdout |     parts = urlsplit(endpoint_url)
gunicorn-registry stdout |   File "/usr/lib64/python3.8/urllib/parse.py", line 423, in urlsplit
gunicorn-registry stdout |     url, scheme, _coerce_result = _coerce_args(url, scheme)
gunicorn-registry stdout |   File "/usr/lib64/python3.8/urllib/parse.py", line 124, in _coerce_args
gunicorn-registry stdout |     return _decode_args(args) + (_encode_result,)
gunicorn-registry stdout |   File "/usr/lib64/python3.8/urllib/parse.py", line 108, in _decode_args
gunicorn-registry stdout |     return tuple(x.decode(encoding, errors) if x else '' for x in args)
gunicorn-registry stdout |   File "/usr/lib64/python3.8/urllib/parse.py", line 108, in <genexpr>
gunicorn-registry stdout |     return tuple(x.decode(encoding, errors) if x else '' for x in args)
gunicorn-registry stdout | AttributeError: 'tuple' object has no attribute 'decode'
gunicorn-registry stdout | 2021-03-25 07:24:31,861 [258] [INFO] [gunicorn.access]  - - [25/Mar/2021:07:24:31 +0000] "POST /v2/qateam/demo2/blobs/uploads/ HTTP/1.1" 500 0 "-" "-"
nginx stdout | 10.131.3.40 () - - [25/Mar/2021:07:24:31 +0000] "POST /v2/qateam/demo2/blobs/uploads/ HTTP/1.1" 500 141 "-" "docker/20.10.5 go/go1.13.15 git-commit/363e9a8 kernel/4.19.121-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5 \x5C(darwin\x5C))" (0.011 1455 0.012)

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

quay_app_350_pod.logs
1.79 MB
2021/03/25 3:32 AM

Quay push image was failed with 500 error code

Details

Description

Attachments

Attachments

Activity

People

Dates