Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-3753

Pushing big layers to Quay deployed on Azure OpenShift Cluster results in a 413

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • quay-v3.8.0
    • quay-v3.6.6
    • quay
    • 0

    Description

      # docker push quay-quay-quay-enterprise.apps.rcegan.azemea.com/ibazulic/test-big-layers:latest
The push refers to repository [quay-quay-quay-enterprise.apps.rcegan.azemea.com/ibazulic/test-big-layers]
bf3b324a82de: Pushing [==================================================>]  2.434GB/2.434GB
e0817fbe1d23: Pushing [======================>                            ]  3.203GB/7.279GB
06074ebe4cc0: Pushed
e7901ff7c751: Pushed
a13c519c6361: Pushed
error parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.14.1</center>\r\n</body>\r\n</html>\r\n"

      Quay is backed by Azure blob storage engine. According to the nginx configuration, we allow up to 20 GB layers to be pushed at once. Since this issue does not happen on AWS, this is something that's inherent to Azure.

      We first see a 499 failure on a patch command:

      nginx stdout | 10.130.2.11 () - - [05/May/2022:10:15:07 +0000] "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/5a78111c-05a1-4b40-9160-2cbb59d6743d HTTP/1.1" 499 0 "-" "docker/20.10.14 go/go1.16.15 git-commit/87a90dc kernel/5.10.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.14 \x5C(linux\x5C))" (295.210 964950573 -)

      This is followed by a failure in the gunicorn-registry worker:

      nginx stdout | 2022/05/05 10:15:47 [error] 106#0: *769 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /v1/_internal_ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v1/_internal_ping", host: "localhost:8080"
...
nginx stdout | 2022/05/05 10:15:48 [error] 106#0: *770 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /v1/_internal_ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v1/_internal_ping", host: "localhost:8080"
...
gunicorn-web stdout | 2022-05-05 10:15:48,788 [222] [WARNING] [health.healthcheck] [FAILED HEALTH CHECK] {'services_expanded': {'registry_gunicorn': {'status': False, 'failure': 'Got non-200 response for worker: 502'}, 'web_gunicorn': {'status': True}, 'service_key': {'status': True}, 'disk_space': {'status': True}, 'database': {'status': True}, 'auth': {'status': True}}, 'notes': [], 'is_testing': False, 'config_provider': 'k8s', 'local_service_key_id': 'I-4Izl8kZruZ4ivvJ-1E-BaYSwe6d_qXDEmZvFOJIo0', 'hostname': 'quay-quay-app-d6556d95c-xg4lc'}
...
gunicorn-web stdout | 2022-05-05 10:15:49,564 [220] [WARNING] [health.healthcheck] [FAILED HEALTH CHECK] {'services_expanded': {'registry_gunicorn': {'status': False, 'failure': 'Got non-200 response for worker: 502'}, 'web_gunicorn': {'status': True}, 'service_key': {'status': True}, 'disk_space': {'status': True}, 'database': {'status': True}, 'auth': {'status': True}}, 'notes': [], 'is_testing': False, 'config_provider': 'k8s', 'local_service_key_id': 'I-4Izl8kZruZ4ivvJ-1E-BaYSwe6d_qXDEmZvFOJIo0', 'hostname': 'quay-quay-app-d6556d95c-xg4lc'}

      The end result is a 413 which seems like a red herring:

      nginx stdout | 2022/05/05 10:15:51 [error] 106#0: *473 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.128.2.6, server: , request: "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855", host: "quay-quay-quay-enterprise.apps.rcegan.azemea.com"
nginx stdout | 2022/05/05 10:15:51 [error] 106#0: *473 client intended to send too large body: 2516758573 bytes, client: 10.128.2.6, server: , request: "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock/v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855", host: "quay-quay-quay-enterprise.apps.rcegan.azemea.com"
nginx stdout | 2022/05/05 10:15:51 [error] 106#0: *739 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.128.2.6, server: , request: "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86", host: "quay-quay-quay-enterprise.apps.rcegan.azemea.com"
nginx stdout | 2022/05/05 10:15:51 [error] 106#0: *739 client intended to send too large body: 92949571 bytes, client: 10.128.2.6, server: , request: "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock/v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86", host: "quay-quay-quay-enterprise.apps.rcegan.azemea.com"
nginx stdout | 10.128.2.6 () - - [05/May/2022:10:15:51 +0000] "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86 HTTP/1.1" 413 199 "-" "docker/20.10.14 go/go1.16.15 git-commit/87a90dc kernel/5.10.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.14 \x5C(linux\x5C))" (38.112 92997279 37.821)
nginx stdout | 10.128.2.6 () - - [05/May/2022:10:15:51 +0000] "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855 HTTP/1.1" 413 199 "-" "docker/20.10.14 go/go1.16.15 git-commit/87a90dc kernel/5.10.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.14 \x5C(linux\x5C))" (339.307 2518083231 338.989)

      Full log is uploaded.

      Seems that the same thing does not happen when Quay is deployed on a standalone VM backed again by Azure storage:

      # docker push 20.67.27.216/ibazulic/test-big-layers:latest
The push refers to repository [20.67.27.216/ibazulic/test-big-layers]
bf3b324a82de: Pushed
e0817fbe1d23: Pushed
06074ebe4cc0: Pushed
e7901ff7c751: Pushed
a13c519c6361: Pushed
latest: digest: sha256:034bf71b04784bc91fee4ed31d2e4321c0f37f165e20d5da2bbae15228db7d38 size: 1378

      Full Quay VM logs are also attached. Can you please check this issue?

      Attachments

        1. image-2022-06-30-16-37-53-142.png
          image-2022-06-30-16-37-53-142.png
          233 kB
        2. image-2022-06-30-16-40-29-033.png
          image-2022-06-30-16-40-29-033.png
          326 kB
        3. quay-app.log
          1.92 MB
        4. quay-vm-standalone.log
          2.08 MB

        Issue Links

          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. PROJQUAY-3999 Pushing big layers to Quay deployed on Azure OpenShift Cluster results in a 413

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          mentioned on

          GitLab Merge request - Updated US source to: 78dd917 storage: optimize large azure chunked upload (PROJQUAY-3753) (#1407)

          Activity

            People

              sleesinc Kenny Lee Sin Cheong
              rhn-support-ibazulic Ivan Bazulic
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: