Loading...

Type: Bug
Resolution: Obsolete
Priority: Minor
Fix Version/s: z-stream
Affects Version/s: quay-v3.6.0
Component/s: quay
Labels:
- quay-enterprise
- triaged

Blocked:
False
Ready:
False
Release Note Text:
undefined
Market:

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description:

This is an issue found when click a canceled build job, the results is getting Quay 500 error page, checked Quay App POD logs, get error "gunicorn-web stdout | KeyError: 'local_us'", see attached Quay App POD logs quay3602_app_pod.logs

Note: Confirmed this issue is not existed in Quay 3.5.6

Quay Image:

quay-operator-bundle-container-v3.6.0-18

Click Canceled Build Job hit quay 500 error page

gunicorn-web stdout | 2021-08-24 03:44:58,592 [221] [ERROR] [gunicorn.error] Error handling request /api/v1/repository/quay/demo/build/cd5d2e36-cc1d-431d-9273-4e66fd917a66/logs?start=0
gunicorn-web stdout | Traceback (most recent call last):
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
gunicorn-web stdout |     self.handle_request(listener_name, req, client, addr)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
gunicorn-web stdout |     super().handle_request(listener_name, req, sock, addr)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request
gunicorn-web stdout |     respiter = self.wsgi(environ, resp.start_response)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2463, in __call__
gunicorn-web stdout |     return self.wsgi_app(environ, start_response)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
gunicorn-web stdout |     return self.app(environ, start_response)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app
gunicorn-web stdout |     response = self.handle_exception(e)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 269, in error_router
gunicorn-web stdout |     return original_handler(e)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1866, in handle_exception
gunicorn-web stdout |     reraise(exc_type, exc_value, tb)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 38, in reraise
gunicorn-web stdout |     raise value.with_traceback(tb)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
gunicorn-web stdout |     response = self.full_dispatch_request()
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request
gunicorn-web stdout |     rv = self.handle_user_exception(e)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 269, in error_router
gunicorn-web stdout |     return original_handler(e)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
gunicorn-web stdout |     return original_handler(e)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
gunicorn-web stdout |     reraise(exc_type, exc_value, tb)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 38, in reraise
gunicorn-web stdout |     raise value.with_traceback(tb)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
gunicorn-web stdout |     rv = self.dispatch_request()
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
gunicorn-web stdout |     return self.view_functions[rule.endpoint](**req.view_args)
gunicorn-web stdout |   File "/quay-registry/endpoints/decorators.py", line 212, in wrapper
gunicorn-web stdout |     return func(*args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/auth/decorators.py", line 65, in wrapper
gunicorn-web stdout |     return func(*args, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/utils/cors.py", line 35, in wrapped_function
gunicorn-web stdout |     resp = make_response(f(*args, **kwargs))
gunicorn-web stdout |   File "/quay-registry/endpoints/csrf.py", line 73, in wrapper
gunicorn-web stdout |     resp = func(*args, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 458, in wrapper
gunicorn-web stdout |     resp = resource(*args, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask/views.py", line 89, in view
gunicorn-web stdout |     return self.dispatch_request(*args, **kwargs)
gunicorn-web stdout |   File "/usr/local/lib/python3.8/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
gunicorn-web stdout |     resp = meth(*args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/decorators.py", line 141, in wrapper
gunicorn-web stdout |     return func(*args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/__init__.py", line 241, in wrapper
gunicorn-web stdout |     return func(namespace, repository, *args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/decorators.py", line 120, in wrapper
gunicorn-web stdout |     return func(*args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/__init__.py", line 295, in wrapped
gunicorn-web stdout |     return func(self, namespace, repository, *args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/__init__.py", line 266, in wrapped
gunicorn-web stdout |     return func(self, namespace_name, repository_name, *args, **kwargs)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/build.py", line 517, in get
gunicorn-web stdout |     return get_logs_or_log_url(build)
gunicorn-web stdout |   File "/quay-registry/endpoints/api/build.py", line 469, in get_logs_or_log_url
gunicorn-web stdout |     "logs_url": log_archive.get_file_url(build.uuid, get_request_ip(), requires_cors=True)
gunicorn-web stdout |   File "/quay-registry/data/userfiles.py", line 121, in get_file_url
gunicorn-web stdout |     url = self._storage.get_direct_download_url(
gunicorn-web stdout |   File "/quay-registry/storage/distributedstorage.py", line 82, in get_direct_download_url
gunicorn-web stdout |     download_url = self._get_direct_download_url(
gunicorn-web stdout |   File "/quay-registry/storage/distributedstorage.py", line 24, in wrapper
gunicorn-web stdout |     storage = self._storages[random.sample(locations, 1)[0]]
gunicorn-web stdout | KeyError: 'local_us'
gunicorn-web stdout | 2021-08-24 03:44:58,595 [221] [INFO] [gunicorn.access]  - - [24/Aug/2021:03:44:58 +0000] "GET /api/v1/repository/quay/demo/build/cd5d2e36-cc1d-431d-9273-4e66fd917a66/logs?start=0 HTTP/1.0" 500 0 "-" "-"
nginx stdout | 10.129.2.17 () - - [24/Aug/2021:03:44:58 +0000] "GET /api/v1/repository/quay/demo/build/cd5d2e36-cc1d-431d-9273-4e66fd917a66/logs?start=0 HTTP/1.1" 500 141 "https://quayv360.apps.quay-perf-738.perfscale.devcluster.openshift.com/repository/quay/demo/build/cd5d2e36-cc1d-431d-9273-4e66fd917a66" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Firefox/91.0" (0.026 1659 0.026)

Config.yaml:

cat config.yaml 
BROWSER_API_CALLS_XHR_ONLY: false
FEATURE_EXTENDED_REPOSITORY_NAMES: true
CREATE_REPOSITORY_ON_PUSH_PUBLIC: true
FEATURE_USER_INITIALIZE: true
SERVER_HOSTNAME: quayv360.apps.quay-perf-738.perfscale.devcluster.openshift.com
ALLOWED_OCI_ARTIFACT_TYPES: 
    application/vnd.cncf.helm.config.v1+json: 
    - application/tar+gzip
    application/vnd.oci.image.layer.v1.tar+gzip+encrypted:
    - application/vnd.oci.image.layer.v1.tar+gzip+encrypted
    application/vnd.oci.image.config.v1+json:
    - application/vnd.oci.image.layer.v1.tar+zstd
    application/vnd.oci.image.config.v1+json:
    - application/vnd.dev.cosign.simplesigning.v1+json
DEFAULT_TAG_EXPIRATION: 4w
TAG_EXPIRATION_OPTIONS:
- 2w
- 4w
- 8w
FEATURE_GENERAL_OCI_SUPPORT: true
FEATURE_HELM_OCI_SUPPORT: false
SUPER_USERS:
  - quay
  - admin
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
  - default
DISTRIBUTED_STORAGE_PREFERENCE:
  - default
DISTRIBUTED_STORAGE_CONFIG:
  default:
    - S3Storage
    - s3_bucket: quay360
      storage_path: /quay360
      s3_access_key: ******
      s3_secret_key: ******
      host: s3.us-east-2.amazonaws.com

Steps:

Deploy Quay with Quay Operator
Use config editor to enable build and config Github trigger
Create new Build Job
Cancel the running Build Job
On Image repo information page, click the canceled build job

Expected Results:

can see the details of canceled job successfully.

Actual Results:

Hit Quay 500 error page.

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

image-2021-08-24-14-37-04-648.png
2021/08/24 6:37 AM
241 kB
luffy zhang
quay3602_app_pod.logs
2021/08/24 6:41 AM
391 kB
luffy zhang

Details

Description

Attachments

Attachments

Easy Agile Planning Poker

Activity

People

Dates