Major Description:
This is an issue found When Quay use NetApp ONTAP S3 Object Storage as unmanaged Storage, try to push image with large layer size, the results was failed, see attached Quay logs quay312_ontap_app_pod1.logs
Quay: quay-operator-bundle-container-v3.12.0-20
podman push quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/qateam/test123 --tls-verify=false --creds quay:password Getting image source signatures Copying blob 2653d992f4ef done | Copying blob 637a3c7a75fa [--------------------------------------] 15.0MiB / 9.4GiB | 225.8 MiB/s Error: writing blob: uploading layer to https://quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/v2/qateam/test123/blobs/uploads/f6739617-28c0-4d04-a981-04a381c730a3?digest=sha256%3Abdd1d5c2c2ed5a2cefd922c68d2e3aa6030377acb45640f972ce735e119cf85b: received unexpected HTTP status: 504 Gateway Time-out podman push quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/qateam/test123 --tls-verify=false --creds quay:password Getting image source signatures Copying blob 2653d992f4ef [=====>--------------------------------] 30.0MiB / 206.5MiB | 183.3 MiB/s Copying blob 637a3c7a75fa [--------------------------------------] 13.0MiB / 9.4GiB | 252.3 MiB/s Error: writing blob: Patch "https://quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/v2/qateam/test123/blobs/uploads/4108c5a7-55c8-4ae6-a465-b1d971b068b9": write tcp 10.0.1.90:34436->52.188.33.235:443: write: broken pipe podman push quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/qateam/test123 --tls-verify=false --creds quay:password Getting image source signatures Copying blob 2653d992f4ef [=====>--------------------------------] 31.0MiB / 206.5MiB | 147.1 MiB/s Copying blob 637a3c7a75fa done | Error: writing blob: Patch "https://quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/v2/qateam/test123/blobs/uploads/e8cfc502-c5b2-4ec8-9fb5-7ffc91caa6af": write tcp 10.0.1.90:42300->52.188.33.235:443: write: broken pipe
Quay Config.yaml:
DISTRIBUTED_STORAGE_CONFIG: local_us: - RadosGWStorage - access_key: ****** bucket_name: bucket1 hostname: poc5113.cpoccloud.com is_secure: true port: 10443 secret_key: ****** storage_path: /datastorage/registry maximum_chunk_size_mb: 32 server_side_assembly: false DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: - local_us DISTRIBUTED_STORAGE_PREFERENCE: - local_us
Quay Logs:
gunicorn-registry stdout | 2024-07-09 07:22:01,619 [247] [ERROR] [gunicorn.error] Error handling request /v2/qateam/test123/blobs/uploads/a5ab25ac-f4f9-41b1-9130-4fa5c0699259 gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn gunicorn-registry stdout | conn = connection.create_connection( gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection gunicorn-registry stdout | raise err gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection gunicorn-registry stdout | sock.connect(sa) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gevent/_socketcommon.py", line 590, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | TimeoutError: [Errno 110] Connection timed out gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/httpsession.py", line 465, in send gunicorn-registry stdout | urllib_response = conn.urlopen( gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 801, in urlopen gunicorn-registry stdout | retries = retries.increment( gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/util/retry.py", line 527, in increment gunicorn-registry stdout | raise six.reraise(type(error), error, _stacktrace) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise gunicorn-registry stdout | raise value gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen gunicorn-registry stdout | httplib_response = self._make_request( gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request gunicorn-registry stdout | self._validate_conn(conn) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn gunicorn-registry stdout | conn.connect() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect gunicorn-registry stdout | self.sock = conn = self._new_conn() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn gunicorn-registry stdout | raise NewConnectionError( gunicorn-registry stdout | urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fec329687f0>: Failed to establish a new connection: [Errno 110] Connection timed out gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle gunicorn-registry stdout | self.handle_request(listener_name, req, client, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 128, in handle_request gunicorn-registry stdout | super().handle_request(listener_name, req, sock, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request gunicorn-registry stdout | respiter = self.wsgi(environ, resp.start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2213, in __call__ gunicorn-registry stdout | return self.wsgi_app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 182, in __call__ gunicorn-registry stdout | return self.app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2193, in wsgi_app gunicorn-registry stdout | response = self.handle_exception(e) ........ gunicorn-registry stdout | return self._emitter.emit(aliased_event_name, **kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/hooks.py", line 256, in emit gunicorn-registry stdout | return self._emit(event_name, kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/hooks.py", line 239, in _emit gunicorn-registry stdout | response = handler(**kwargs) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 207, in __call__ gunicorn-registry stdout | if self._checker(**checker_kwargs): gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 284, in __call__ gunicorn-registry stdout | should_retry = self._should_retry( gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 320, in _should_retry gunicorn-registry stdout | return self._checker(attempt_number, response, caught_exception) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 363, in __call__ gunicorn-registry stdout | checker_response = checker( gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 247, in __call__ gunicorn-registry stdout | return self._check_caught_exception( gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception gunicorn-registry stdout | raise caught_exception gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/endpoint.py", line 281, in _do_get_response gunicorn-registry stdout | http_response = self._send(request) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/endpoint.py", line 377, in _send gunicorn-registry stdout | return self.http_session.send(request) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/botocore/httpsession.py", line 494, in send gunicorn-registry stdout | raise EndpointConnectionError(endpoint_url=request.url, error=e) gunicorn-registry stdout | botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://poc5113.cpoccloud.com:10443/bucket1/datastorage/registry/uploads/027025a1-9c34-44c1-b68a-c78178de4c50?uploads" gunicorn-registry stdout | 2024-07-09 07:22:01,621 [247] [INFO] [gunicorn.access] - - [09/Jul/2024:07:22:01 +0000] "PATCH /v2/qateam/test123/blobs/uploads/a5ab25ac-f4f9-41b1-9130-4fa5c0699259 HTTP/1.1" 500 0 "-" "-" nginx stdout | 10.130.2.21 (-) - - [09/Jul/2024:07:22:01 +0000] "PATCH /v2/qateam/test123/blobs/uploads/a5ab25ac-f4f9-41b1-9130-4fa5c0699259 HTTP/1.1" 500 141 "-" "containers/5.29.2 (github.com/containers/image)" (664.831 182000 664.831)
