Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-4027

Quay 3.7.3 push image was failed when image layer size is more than 30GB

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • quay-v3.6.8, quay-v3.7.3
    • -area/registry, quay
    • False
    • None
    • False

      Description:

      This is an issue found when push image to quay, if the size of image layer is more than 30GB, the results is pushing image was failed with error message "Error: Error copying image to the remote destination: Error writing blob: Error determining upload URL: http: no Location header in response", while if pushing another image with the size of image layer is 9GB, it works well. So the expected behavior is if quay doesn't support single layer more than 30GB, then should give correct error message to quay users. refer to Quay POD logs quay368_app_pod2.logs 

      Quay Image: quay-operator-bundle-container-v3.6.8-9

      Note: The backend storage of quay is azure blob storage.

      [root@ip-10-0-1-110 centos]# podman push demo-quay-quay368.apps.quaytest-8960.qe.azure.devcluster.openshift.com/quay/demo:newimage39GB --tls-verify=false --creds quay:password
      Getting image source signatures
      Copying blob 9dc780ebe92b [=============>------------------------] 14.3GiB / 39.0GiB
      Copying blob a790f937a6ae [==============================>-------] 63.2MiB / 76.6MiB
      
      [root@ip-10-0-1-110 centos]# podman push demo-quay-quay368.apps.quaytest-8960.qe.azure.devcluster.openshift.com/quay/demo:newimage39GB --tls-verify=false --creds quay:password
      Getting image source signatures
      Copying blob 9dc780ebe92b [===========================>----------] 28.4GiB / 39.0GiB
      Copying blob a790f937a6ae [==============================>-------] 63.2MiB / 76.6MiB
      
      [root@ip-10-0-1-110 centos]# podman push demo-quay-quay368.apps.quaytest-8960.qe.azure.devcluster.openshift.com/quay/demo:newimage39GB --tls-verify=false --creds quay:password
      Getting image source signatures
      Copying blob 9dc780ebe92b done
      Copying blob a790f937a6ae done
      Error: Error copying image to the remote destination: Error writing blob: Error determining upload URL: http: no Location header in response 

      Quay POD logs:

      gunicorn-registry stdout | 2022-06-30 09:32:57,828 [230] [DEBUG] [urllib3.connectionpool] https://quay37390.blob.core.windows.net:443 "PUT /quay368/datastorage/registry/uploads/fb1e8042-3d06-459b-8498-afd6d2de6535?blockid=MDAwMDA%3D&comp=block HTTP/1.1" 201 0
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 201
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response headers:
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy]     'Content-Length': '0'
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy]     'Content-MD5': 'REDACTED'
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy]     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy]     'x-ms-request-id': 'ab7acfb1-901e-0036-2664-8c818d000000'
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy]     'x-ms-client-request-id': '9dc301dc-f857-11ec-8044-0a580a80022f'
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy]     'x-ms-version': 'REDACTED'
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy]     'x-ms-request-server-encrypted': 'REDACTED'
      gunicorn-registry stdout | 2022-06-30 09:32:57,829 [230] [INFO] [azure.core.pipeline.policies.http_logging_policy]     'Date': 'Thu, 30 Jun 2022 09:32:56 GMT'
      nginx stdout | 10.129.2.16 () - - [30/Jun/2022:09:32:57 +0000] "PATCH /v2/quay/demo/blobs/uploads/fb1e8042-3d06-459b-8498-afd6d2de6535 HTTP/1.1" 400 0 "-" "libpod/1.6.4" (382.438 135360953 -) 

              Unassigned Unassigned
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: