Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-7462

Quay push image with large layer size was failed when using NetApp ONTAP S3 Object Storage

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • quay-v3.11.0, quay-v3.12.0, quay-v3.12.1
    • quay
    • False
    • None
    • False

      Description:

      This is an issue found When Quay use NetApp ONTAP S3 Object Storage as unmanaged Storage, try to push image with large layer size, the results was failed, see attached Quay logs quay312_ontap_app_pod1.logs 

      Quay:  quay-operator-bundle-container-v3.12.0-20

      podman push quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/qateam/test123 --tls-verify=false --creds quay:password
      Getting image source signatures
      Copying blob 2653d992f4ef done   | 
      Copying blob 637a3c7a75fa [--------------------------------------] 15.0MiB / 9.4GiB | 225.8 MiB/s
      Error: writing blob: uploading layer to https://quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/v2/qateam/test123/blobs/uploads/f6739617-28c0-4d04-a981-04a381c730a3?digest=sha256%3Abdd1d5c2c2ed5a2cefd922c68d2e3aa6030377acb45640f972ce735e119cf85b: received unexpected HTTP status: 504 Gateway Time-out
      
      podman push quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/qateam/test123 --tls-verify=false --creds quay:password
      Getting image source signatures
      Copying blob 2653d992f4ef [=====>--------------------------------] 30.0MiB / 206.5MiB | 183.3 MiB/s
      Copying blob 637a3c7a75fa [--------------------------------------] 13.0MiB / 9.4GiB | 252.3 MiB/s
      Error: writing blob: Patch "https://quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/v2/qateam/test123/blobs/uploads/4108c5a7-55c8-4ae6-a465-b1d971b068b9": write tcp 10.0.1.90:34436->52.188.33.235:443: write: broken pipe
      
      podman push quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/qateam/test123 --tls-verify=false --creds quay:password 
      Getting image source signatures 
      Copying blob 2653d992f4ef [=====>--------------------------------] 31.0MiB / 206.5MiB | 147.1 MiB/s 
      Copying blob 637a3c7a75fa done   |  
      Error: writing blob: Patch "https://quayontap-quay-quay312.apps.quaytest-2752.qe.azure.devcluster.openshift.com/v2/qateam/test123/blobs/uploads/e8cfc502-c5b2-4ec8-9fb5-7ffc91caa6af": write tcp 10.0.1.90:42300->52.188.33.235:443: write: broken pipe
      

      Quay Config.yaml:

      DISTRIBUTED_STORAGE_CONFIG:
        local_us:
        - RadosGWStorage
        - access_key: ******
          bucket_name: bucket1
          hostname: poc5113.cpoccloud.com
          is_secure: true
          port: 10443
          secret_key: ******
          storage_path: /datastorage/registry
          maximum_chunk_size_mb: 32
          server_side_assembly: false
      DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
      - local_us
      DISTRIBUTED_STORAGE_PREFERENCE:
      - local_us

      Quay Logs:

      gunicorn-registry stdout | 2024-07-09 07:22:01,619 [247] [ERROR] [gunicorn.error] Error handling request /v2/qateam/test123/blobs/uploads/a5ab25ac-f4f9-41b1-9130-4fa5c0699259
      gunicorn-registry stdout | Traceback (most recent call last):
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
      gunicorn-registry stdout |     conn = connection.create_connection(
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
      gunicorn-registry stdout |     raise err
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
      gunicorn-registry stdout |     sock.connect(sa)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gevent/_socketcommon.py", line 590, in connect
      gunicorn-registry stdout |     self._internal_connect(address)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect
      gunicorn-registry stdout |     raise _SocketError(err, strerror(err))
      gunicorn-registry stdout | TimeoutError: [Errno 110] Connection timed out
      gunicorn-registry stdout | During handling of the above exception, another exception occurred:
      gunicorn-registry stdout | Traceback (most recent call last):
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/httpsession.py", line 465, in send
      gunicorn-registry stdout |     urllib_response = conn.urlopen(
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 801, in urlopen
      gunicorn-registry stdout |     retries = retries.increment(
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/util/retry.py", line 527, in increment
      gunicorn-registry stdout |     raise six.reraise(type(error), error, _stacktrace)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
      gunicorn-registry stdout |     raise value
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen
      gunicorn-registry stdout |     httplib_response = self._make_request(
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request
      gunicorn-registry stdout |     self._validate_conn(conn)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn
      gunicorn-registry stdout |     conn.connect()
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect
      gunicorn-registry stdout |     self.sock = conn = self._new_conn()
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
      gunicorn-registry stdout |     raise NewConnectionError(
      gunicorn-registry stdout | urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7fec329687f0>: Failed to establish a new connection: [Errno 110] Connection timed out
      gunicorn-registry stdout | During handling of the above exception, another exception occurred:
      gunicorn-registry stdout | Traceback (most recent call last):
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle
      gunicorn-registry stdout |     self.handle_request(listener_name, req, client, addr)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 128, in handle_request
      gunicorn-registry stdout |     super().handle_request(listener_name, req, sock, addr)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request
      gunicorn-registry stdout |     respiter = self.wsgi(environ, resp.start_response)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 2213, in __call__
      gunicorn-registry stdout |     return self.wsgi_app(environ, start_response)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 182, in __call__
      gunicorn-registry stdout |     return self.app(environ, start_response)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/flask/app.py", line 2193, in wsgi_app
      gunicorn-registry stdout |     response = self.handle_exception(e) 
      ........
      gunicorn-registry stdout |     return self._emitter.emit(aliased_event_name, **kwargs)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/hooks.py", line 256, in emit
      gunicorn-registry stdout |     return self._emit(event_name, kwargs)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/hooks.py", line 239, in _emit
      gunicorn-registry stdout |     response = handler(**kwargs)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 207, in __call__
      gunicorn-registry stdout |     if self._checker(**checker_kwargs):
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 284, in __call__
      gunicorn-registry stdout |     should_retry = self._should_retry(
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 320, in _should_retry
      gunicorn-registry stdout |     return self._checker(attempt_number, response, caught_exception)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 363, in __call__
      gunicorn-registry stdout |     checker_response = checker(
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 247, in __call__
      gunicorn-registry stdout |     return self._check_caught_exception(
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
      gunicorn-registry stdout |     raise caught_exception
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/endpoint.py", line 281, in _do_get_response
      gunicorn-registry stdout |     http_response = self._send(request)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/endpoint.py", line 377, in _send
      gunicorn-registry stdout |     return self.http_session.send(request)
      gunicorn-registry stdout |   File "/app/lib/python3.9/site-packages/botocore/httpsession.py", line 494, in send
      gunicorn-registry stdout |     raise EndpointConnectionError(endpoint_url=request.url, error=e)
      gunicorn-registry stdout | botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://poc5113.cpoccloud.com:10443/bucket1/datastorage/registry/uploads/027025a1-9c34-44c1-b68a-c78178de4c50?uploads"
      gunicorn-registry stdout | 2024-07-09 07:22:01,621 [247] [INFO] [gunicorn.access]  - - [09/Jul/2024:07:22:01 +0000] "PATCH /v2/qateam/test123/blobs/uploads/a5ab25ac-f4f9-41b1-9130-4fa5c0699259 HTTP/1.1" 500 0 "-" "-"
      nginx stdout | 10.130.2.21 (-) - - [09/Jul/2024:07:22:01 +0000] "PATCH /v2/qateam/test123/blobs/uploads/a5ab25ac-f4f9-41b1-9130-4fa5c0699259 HTTP/1.1" 500 141 "-" "containers/5.29.2 (github.com/containers/image)" (664.831 182000 664.831)

              rhn-support-ibazulic Ivan Bazulic
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: