-
Bug
-
Resolution: Done
-
Major
-
None
-
quay-v3.7.2
# docker push quay-quay-quay-enterprise.apps.rcegan.azemea.com/ibazulic/test-big-layers:latest The push refers to repository [quay-quay-quay-enterprise.apps.rcegan.azemea.com/ibazulic/test-big-layers] bf3b324a82de: Pushing [==================================================>] 2.434GB/2.434GB e0817fbe1d23: Pushing [======================> ] 3.203GB/7.279GB 06074ebe4cc0: Pushed e7901ff7c751: Pushed a13c519c6361: Pushed error parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.14.1</center>\r\n</body>\r\n</html>\r\n"
Quay is backed by Azure blob storage engine. According to the nginx configuration, we allow up to 20 GB layers to be pushed at once. Since this issue does not happen on AWS, this is something that's inherent to Azure.
We first see a 499 failure on a patch command:
nginx stdout | 10.130.2.11 () - - [05/May/2022:10:15:07 +0000] "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/5a78111c-05a1-4b40-9160-2cbb59d6743d HTTP/1.1" 499 0 "-" "docker/20.10.14 go/go1.16.15 git-commit/87a90dc kernel/5.10.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.14 \x5C(linux\x5C))" (295.210 964950573 -)
This is followed by a failure in the gunicorn-registry worker:
nginx stdout | 2022/05/05 10:15:47 [error] 106#0: *769 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /v1/_internal_ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v1/_internal_ping", host: "localhost:8080" ... nginx stdout | 2022/05/05 10:15:48 [error] 106#0: *770 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /v1/_internal_ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v1/_internal_ping", host: "localhost:8080" ... gunicorn-web stdout | 2022-05-05 10:15:48,788 [222] [WARNING] [health.healthcheck] [FAILED HEALTH CHECK] {'services_expanded': {'registry_gunicorn': {'status': False, 'failure': 'Got non-200 response for worker: 502'}, 'web_gunicorn': {'status': True}, 'service_key': {'status': True}, 'disk_space': {'status': True}, 'database': {'status': True}, 'auth': {'status': True}}, 'notes': [], 'is_testing': False, 'config_provider': 'k8s', 'local_service_key_id': 'I-4Izl8kZruZ4ivvJ-1E-BaYSwe6d_qXDEmZvFOJIo0', 'hostname': 'quay-quay-app-d6556d95c-xg4lc'} ... gunicorn-web stdout | 2022-05-05 10:15:49,564 [220] [WARNING] [health.healthcheck] [FAILED HEALTH CHECK] {'services_expanded': {'registry_gunicorn': {'status': False, 'failure': 'Got non-200 response for worker: 502'}, 'web_gunicorn': {'status': True}, 'service_key': {'status': True}, 'disk_space': {'status': True}, 'database': {'status': True}, 'auth': {'status': True}}, 'notes': [], 'is_testing': False, 'config_provider': 'k8s', 'local_service_key_id': 'I-4Izl8kZruZ4ivvJ-1E-BaYSwe6d_qXDEmZvFOJIo0', 'hostname': 'quay-quay-app-d6556d95c-xg4lc'}
The end result is a 413 which seems like a red herring:
nginx stdout | 2022/05/05 10:15:51 [error] 106#0: *473 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.128.2.6, server: , request: "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855", host: "quay-quay-quay-enterprise.apps.rcegan.azemea.com" nginx stdout | 2022/05/05 10:15:51 [error] 106#0: *473 client intended to send too large body: 2516758573 bytes, client: 10.128.2.6, server: , request: "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock/v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855", host: "quay-quay-quay-enterprise.apps.rcegan.azemea.com" nginx stdout | 2022/05/05 10:15:51 [error] 106#0: *739 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.128.2.6, server: , request: "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86", host: "quay-quay-quay-enterprise.apps.rcegan.azemea.com" nginx stdout | 2022/05/05 10:15:51 [error] 106#0: *739 client intended to send too large body: 92949571 bytes, client: 10.128.2.6, server: , request: "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock/v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86", host: "quay-quay-quay-enterprise.apps.rcegan.azemea.com" nginx stdout | 10.128.2.6 () - - [05/May/2022:10:15:51 +0000] "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/8c27a1e5-8725-40a2-a184-f4e11fe5bc86 HTTP/1.1" 413 199 "-" "docker/20.10.14 go/go1.16.15 git-commit/87a90dc kernel/5.10.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.14 \x5C(linux\x5C))" (38.112 92997279 37.821) nginx stdout | 10.128.2.6 () - - [05/May/2022:10:15:51 +0000] "PATCH /v2/ibazulic/test-big-layers/blobs/uploads/a1836993-a72f-466e-acde-71c412576855 HTTP/1.1" 413 199 "-" "docker/20.10.14 go/go1.16.15 git-commit/87a90dc kernel/5.10.0-13-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.14 \x5C(linux\x5C))" (339.307 2518083231 338.989)
Full log is uploaded.
Seems that the same thing does not happen when Quay is deployed on a standalone VM backed again by Azure storage:
# docker push 20.67.27.216/ibazulic/test-big-layers:latest
The push refers to repository [20.67.27.216/ibazulic/test-big-layers]
bf3b324a82de: Pushed
e0817fbe1d23: Pushed
06074ebe4cc0: Pushed
e7901ff7c751: Pushed
a13c519c6361: Pushed
latest: digest: sha256:034bf71b04784bc91fee4ed31d2e4321c0f37f165e20d5da2bbae15228db7d38 size: 1378
Full Quay VM logs are also attached. Can you please check this issue?
- clones
-
-
- Closed
-
