Critical Description:
This is an issue found in Quay 3.13.1, when trying to push image with large layer to Quay, hit 500 error code, checked Quay App POD logs, get error "An error occurred (NoSuchUpload) when calling the CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.", pls review this issue, see attached Quay App POD logs,
Quay: quay-operator-bundle-container-v3.13.1-5
Test Image: quay.io/quay-qetest/largesize:latest
podman push quay313hcpv97-quay-quay313.apps.quaytest-3305.qe.azure.devcluster.openshift.com/quayqe/largesizeimage --tls-verify=false --creds quay:password
Getting image source signatures
Copying blob 637a3c7a75fa done |
Copying blob bdd1d5c2c2ed skipped: already exists
Error: writing blob: uploading layer chunked: received unexpected HTTP status: 500 Internal Server Error
Quay config.yaml:
DEFAULT_TAG_EXPIRATION: 2w
DISTRIBUTED_STORAGE_CONFIG:
local_us:
- RadosGWStorage
- access_key: ***
bucket_name: redhat
hostname: partner.hcpdemo.hitachivantara.com
is_secure: true
secret_key: ***
storage_path: /datastorage/registry
signature_version: v4
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
- local_us
DISTRIBUTED_STORAGE_PREFERENCE:
- local_us
PERMANENTLY_DELETE_TAGS: true
RESET_CHILD_MANIFEST_EXPIRATION: true
FEATURE_EXTENDED_REPOSITORY_NAMES: true
CREATE_NAMESPACE_ON_PUSH: true
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_PROXY_CACHE: true
FEATURE_USER_INITIALIZE: true
FEATURE_PROXY_STORAGE: true
FEATURE_UI_V2: true
FEATURE_SUPERUSERS_FULL_ACCESS: true
FEATURE_AUTO_PRUNE: true
Quay Logs:
gunicorn-registry stdout | 2024-11-11 10:00:54,913 [252] [ERROR] [storage.cloud] Exception trying to perform action <bound method ResourceFactory._create_action.<locals>.do_action of s3.MultipartUpload(bucket_name='redhat', object_key='datastorage/registry/uploads/9f4f60f6-bfd7-47af-badc-dc0e4507ba2d', id='110804220604801')> gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/storage/cloud.py", line 607, in _perform_action_with_retry gunicorn-registry stdout | return action(*args, **kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/boto3/resources/factory.py", line 580, in do_action gunicorn-registry stdout | response = action(self, *args, **kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__ gunicorn-registry stdout | response = getattr(parent.meta.client, operation_name)(*args, **params) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call gunicorn-registry stdout | return self._make_api_call(operation_name, kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/client.py", line 980, in _make_api_call gunicorn-registry stdout | raise error_class(parsed_response, operation_name) gunicorn-registry stdout | botocore.errorfactory.NoSuchUpload: An error occurred (NoSuchUpload) when calling the CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. gunicorn-registry stdout | 2024-11-11 10:00:54,914 [252] [ERROR] [gunicorn.error] Error handling request /v2/quayqe/largesizeimage/blobs/uploads/d5746282-c0cb-476b-a215-2820532de8eb gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle gunicorn-registry stdout | self.handle_request(listener_name, req, client, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 128, in handle_request gunicorn-registry stdout | super().handle_request(listener_name, req, sock, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request gunicorn-registry stdout | respiter = self.wsgi(environ, resp.start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2213, in __call__ gunicorn-registry stdout | return self.wsgi_app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__ gunicorn-registry stdout | return self.app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2193, in wsgi_app gunicorn-registry stdout | response = self.handle_exception(e) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2190, in wsgi_app gunicorn-registry stdout | response = self.full_dispatch_request() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1486, in full_dispatch_request gunicorn-registry stdout | rv = self.handle_user_exception(e) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1484, in full_dispatch_request gunicorn-registry stdout | rv = self.dispatch_request() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1469, in dispatch_request gunicorn-registry stdout | return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 253, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 91, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/auth/registry_jwt_auth.py", line 175, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/__init__.py", line 222, in wrapped gunicorn-registry stdout | return func(namespace_name, repo_name, *args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 164, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 189, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 214, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/blob.py", line 380, in upload_chunk gunicorn-registry stdout | _upload_chunk(uploader) ..... gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/boto3/resources/factory.py", line 580, in do_action gunicorn-registry stdout | response = action(self, *args, **kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__ gunicorn-registry stdout | response = getattr(parent.meta.client, operation_name)(*args, **params) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call gunicorn-registry stdout | return self._make_api_call(operation_name, kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/client.py", line 980, in _make_api_call gunicorn-registry stdout | raise error_class(parsed_response, operation_name) gunicorn-registry stdout | botocore.errorfactory.NoSuchUpload: An error occurred (NoSuchUpload) when calling the CompleteMultipartUpload operation: The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. gunicorn-registry stdout | 2024-11-11 10:00:54,915 [252] [INFO] [gunicorn.access] - - [11/Nov/2024:10:00:54 +0000] "PATCH /v2/quayqe/largesizeimage/blobs/uploads/d5746282-c0cb-476b-a215-2820532de8eb HTTP/1.1" 500 0 "-" "-"