Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-7574

Quay push image with large layer size hit 400 error code on Hitachi HCP V9.7

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • quay-v3.12.3
    • quay-v3.12.1
    • quay
    • False
    • None
    • False

      Description:

      When push image to Quay with the size of single image layer size is more than 8GB, quay report 400 error code "PATCH /v2/quayorg/quayrepo/blobs/uploads/1f49a420-fc5c-4465-ab88-9d36b3258d47 HTTP/1.1" 400 "-" "containers/5.24.1 ", pls review attached detailed log

      Note: Storage is Hitachi HCP V9.7

      Quay Image: quay-operator-bundle-container-v3.12.1-3

      $ podman push quayregistry-quay-quay-enterprise-14844.apps.quaytest-14844.qe.devcluster.openshift.com/quayorg/quayrepo:largesize  --tls-verify=false
Getting image source signatures
Copying blob 637a3c7a75fa [=>------------------------------------] 586.0MiB / 9.4GiB
Copying blob 2653d992f4ef done 
WARN[0044] Failed, retrying in 1s ... (1/3). Error: writing blob: Patch "https://quayregistry-quay-quay-enterprise-14844.apps.quaytest-14844.qe.devcluster.openshift.com/v2/quayorg/quayrepo/blobs/uploads/a5b4f60e-8104-4059-80f7-937dd0a125f1": EOF 
Getting image source signatures
Copying blob 637a3c7a75fa [=>------------------------------------] 563.0MiB / 9.4GiB
Copying blob 2653d992f4ef done 
WARN[0085] Failed, retrying in 1s ... (2/3). Error: writing blob: Patch "https://quayregistry-quay-quay-enterprise-14844.apps.quaytest-14844.qe.devcluster.openshift.com/v2/quayorg/quayrepo/blobs/uploads/d4e53b33-74cc-4262-b5f2-db7cc7574baa": EOF 
Getting image source signatures
Copying blob 637a3c7a75fa [=>------------------------------------] 565.0MiB / 9.4GiB
Copying blob 2653d992f4ef done 
WARN[0150] Failed, retrying in 1s ... (3/3). Error: writing blob: Patch "https://quayregistry-quay-quay-enterprise-14844.apps.quaytest-14844.qe.devcluster.openshift.com/v2/quayorg/quayrepo/blobs/uploads/e6835b77-a18d-4af7-ada5-f06d08242396": write tcp 10.0.109.232:54150->3.133.198.230:443: write: connection reset by peer 
Getting image source signatures
Copying blob 637a3c7a75fa [=>------------------------------------] 550.0MiB / 9.4GiB
Copying blob bdd1d5c2c2ed skipped: already exists 
Error: writing blob: Patch "https://quayregistry-quay-quay-enterprise-14844.apps.quaytest-14844.qe.devcluster.openshift.com/v2/quayorg/quayrepo/blobs/uploads/859696f1-8fae-4eed-a301-c9505482310f": write tcp 10.0.109.232:54170->3.133.198.230:443: write: connection reset by peer

      Quay APP POD Logs:

      gunicorn-registry stdout | 2024-08-02 05:37:43,583 [246] [INFO] [gunicorn.access] 10.131.0.30 - - [02/Aug/2024:05:37:43 +0000] "POST /v2/quayorg/quayrepo/blobs/uploads/ HTTP/1.1" 202 0 "-" "containers/5.24.1 (github.com/containers/image)"
gunicorn-registry stdout | 2024-08-02 05:37:45,455 [251] [ERROR] [data.registry_model.blobuploader] storage.stream_upload_chunk returned error No more data after: None
gunicorn-registry stdout | 2024-08-02 05:37:45,455 [251] [ERROR] [endpoints.v2.blob] Exception when uploading blob to 1f49a420-fc5c-4465-ab88-9d36b3258d47
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/blob.py", line 555, in _upload_chunk
gunicorn-registry stdout |     blob_uploader.upload_chunk(app.config, input_fp, start_offset, length)
gunicorn-registry stdout |   File "/quay-registry/data/registry_model/blobuploader.py", line 234, in upload_chunk
gunicorn-registry stdout |     raise BlobUploadException(upload_error)
gunicorn-registry stdout | data.registry_model.blobuploader.BlobUploadException: No more data after: None
gunicorn-registry stdout | 2024-08-02 05:37:45,457 [251] [INFO] [gunicorn.access] 10.131.0.30 - - [02/Aug/2024:05:37:45 +0000] "PATCH /v2/quayorg/quayrepo/blobs/uploads/1f49a420-fc5c-4465-ab88-9d36b3258d47 HTTP/1.1" 400 0 "-" "containers/5.24.1 (github.com/containers/image)"

       

      Quay config.yaml:

      BROWSER_API_CALLS_XHR_ONLY: false
PERMANENTLY_DELETE_TAGS: true
RESET_CHILD_MANIFEST_EXPIRATION: true
FEATURE_EXTENDED_REPOSITORY_NAMES: true
CREATE_NAMESPACE_ON_PUSH: true
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_PROXY_CACHE: true
FEATURE_USER_INITIALIZE: true
FEATURE_PROXY_STORAGE: false
IGNORE_UNKNOWN_MEDIATYPES: true
FEATURE_UI_V2: true
FEATURE_SUPERUSERS_FULL_ACCESS: true
FEATURE_AUTO_PRUNE: true
SUPER_USERS:
  - quay
  - admin
GLOBAL_READONLY_SUPER_USERS:
  - superglobalro
DISTRIBUTED_STORAGE_CONFIG:
  local_us:
  - S3Storage
  - s3_access_key: xxx
    s3_bucket: redhat
    host: partner.hcpdemo.hitachivantara.com
    s3_secret_key: xxx
    storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS:
- local_us
DISTRIBUTED_STORAGE_PREFERENCE:
- local_us
DEFAULT_TAG_EXPIRATION: "30s"
TAG_EXPIRATION_OPTIONS:
- 2w
- 10h
- 30m
- 30s

       

      Note:

      with RadosGWStorage Storage engine, it returns similar error:

      $ podman push quay.io/quay-qetest/largesize quayregistry-quay-quay-enterprise-14839.apps.quaytest-14839.qe.devcluster.openshift.com/user1org/user1repo:largesize --tls-verify=false
Getting image source signatures
Copying blob 637a3c7a75fa [=>------------------------------------] 521.0MiB / 9.4GiB
Copying blob 2653d992f4ef done  WARN[0039] Failed, retrying in 1s ... (1/3). Error: writing blob: Patch "https://quayregistry-quay-quay-enterprise-14839.apps.quaytest-14839.qe.devcluster.openshift.com/v2/user1org/user1repo/blobs/uploads/5840ea1f-25d0-4088-b405-3dcc97d85e85": EOF Getting image source signaturesCopying blob 637a3c7a75fa [=>------------------------------------] 505.0MiB / 9.4GiBCopying blob 637a3c7a75fa [=>------------------------------------] 510.0MiB / 9.4GiBCopying blob 637a3c7a75fa [=>------------------------------------] 512.0MiB / 9.4GiBCopying blob 637a3c7a75fa [=>------------------------------------] 538.0MiB / 9.4GiB
Copying blob 2653d992f4ef done  WARN[0086] Failed, retrying in 1s ... (2/3). Error: writing blob: Patch "https://quayregistry-quay-quay-enterprise-14839.apps.quaytest-14839.qe.devcluster.openshift.com/v2/user1org/user1repo/blobs/uploads/73c633af-b5e8-4a56-b839-0e3afe088531": EOF Getting image source signaturesCopying blob 637a3c7a75fa [=>------------------------------------] 484.0MiB / 9.4GiB
Copying blob 2653d992f4ef done  WARN[0122] Failed, retrying in 1s ... (3/3). Error: writing blob: Patch "https://quayregistry-quay-quay-enterprise-14839.apps.quaytest-14839.qe.devcluster.openshift.com/v2/user1org/user1repo/blobs/uploads/96e15f7f-83c9-4e02-860d-d8ca160024b8": EOF Getting image source signaturesCopying blob 2653d992f4ef done  
Copying blob 637a3c7a75fa [=>------------------------------------] 549.0MiB / 9.4GiBError: writing blob: Patch "https://quayregistry-quay-quay-enterprise-14839.apps.quaytest-14839.qe.devcluster.openshift.com/v2/user1org/user1repo/blobs/uploads/3960ad96-abc7-4576-970f-68a1855a2189": EOF


Quay log:

nginx stdout | 10.128.2.25 (-) - - [26/Jul/2024:03:35:35 +0000] "PATCH /v2/user1org/user1repo/blobs/uploads/f7b24ae0-b30e-4df7-837a-74af7ef3dd0c HTTP/1.1" 400 0 "-" "containers/5.24.1 (github.com/containers/image)" (26.996 46943804 26.996)gunicorn-registry stdout | 2024-07-26 03:35:35,690 [248] [WARNING] [storage.cloud] Error when writing to stream in stream_write_internal at path uploads/2f6a5a20-4237-4fd9-9982-c6a8cf6cba8e: No more data after: Nonegunicorn-registry stdout | 2024-07-26 03:35:37,861 [248] [DEBUG] [boto3.resources.factory] Loading s3:Objectgunicorn-registry stdout | 2024-07-26 03:35:37,862 [248] [DEBUG] [botocore.hooks] Event creating-resource-class.s3.Object: calling handler <function lazy_call.<locals>._handler at 0x7f768dfd39d0>gunicorn-registry stdout | 2024-07-26 03:35:37,862 [248] [ERROR] [data.registry_model.blobuploader] storage.stream_upload_chunk returned error No more data after: Nonegunicorn-registry stdout | 2024-07-26 03:35:37,862 [248] [ERROR] [endpoints.v2.blob] Exception when uploading blob to f7b24ae0-b30e-4df7-837a-74af7ef3dd0cgunicorn-registry stdout | Traceback (most recent call last):gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/blob.py", line 555, in _upload_chunkgunicorn-registry stdout |     blob_uploader.upload_chunk(app.config, input_fp, start_offset, length)gunicorn-registry stdout |   File "/quay-registry/data/registry_model/blobuploader.py", line 234, in upload_chunkgunicorn-registry stdout |     raise BlobUploadException(upload_error)gunicorn-registry stdout | data.registry_model.blobuploader.BlobUploadException: No more data after: Nonegunicorn-registry stdout | 2024-07-26 03:35:37,863 [248] [DEBUG] [endpoints.v2] sending response: b'{"errors":[{"code":"BLOB_UPLOAD_INVALID","detail":{},"message":"blob upload invalid"}]}\n'

       

        1. quayapp1.log
          4.17 MB
          Sean Zhao
        2. quayapp2.log
          4.15 MB
          Sean Zhao

              lzha1981 luffy zhang
              szhao@redhat.com Sean Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: