-
Bug
-
Resolution: Done
-
Critical
-
quay-v3.4.0
Description:
This is an issue found when configuring Quay's backend storage as Azure Blob Storage, push image was failed, checked quay POD Logs ,get "client intended to send too large body"
Note: with the same quay image "quay.io/projectquay/quay:50ba6130", this issue can not be reproduced when quay's backend storage is AWS S3
OCP ENV:
oc get pod demo-quayecosystem-quay-66f6f8c65d-bhmcp -o json | jq '.spec.containers[0].image' "quay.io/projectquay/quay:50ba6130" oc get pod NAME READY STATUS RESTARTS AGE demo-quayecosystem-clair-7fb8959fff-b7ztl 1/1 Running 0 4h12m demo-quayecosystem-clair-postgresql-85f585678c-pt2j2 1/1 Running 0 4h13m demo-quayecosystem-quay-66f6f8c65d-bhmcp 1/1 Running 0 4h15m demo-quayecosystem-quay-config-6ff66f9cc4-55fv4 1/1 Running 0 4h17m demo-quayecosystem-quay-postgresql-9446f4d9-mjhlq 1/1 Running 0 4h19m demo-quayecosystem-redis-566b9ddc5d-vj94f 1/1 Running 0 4h20m quay-operator-5c8f5f8b45-k67v8 1/1 Running 0 4h21m oc get route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD demo-quayecosystem-quay demo-quayecosystem-quay-quay-enterprise.apps.quay-286.qe.azure.devcluster.openshift.com demo-quayecosystem-quay 8443 passthrough/Redirect None demo-quayecosystem-quay-config demo-quayecosystem-quay-config-quay-enterprise.apps.quay-286.qe.azure.devcluster.openshift.com demo-quayecosystem-quay-config 8443 passthrough/Redirect None
With Podman:
[root@ip-10-0-1-223 centos]# podman push docker.io/library/nginx --tls-verify=false --creds=quay:password demo-quayecosystem-quay-quay-enterprise.apps.quay-286.qe.azure.devcluster.openshift.com/qateam/demo
Getting image source signatures
Copying blob d0f104dc0a1f [=============================>--------] 54.0MiB / 69.1MiB
Copying blob ff73b8119c50 [======================================] 60.8MiB / 61.2MiB
Copying blob 6d196faff4ee skipped: already exists
Copying blob ed39597d1556 skipped: already exists
Copying blob ac078d8f08c6 skipped: already exists
Error: Error copying image to the remote destination: Error writing blob: Error determining upload URL: http: no Location header in response
With Docker:
[root@registry lzha]# docker push demo-quayecosystem-quay-quay-enterprise.apps.quay-286.qe.azure.devcluster.openshift.com/qateam/demoThe push refers to a repository [demo-quayecosystem-quay-quay-enterprise.apps.quay-286.qe.azure.devcluster.openshift.com/qateam/demo]98b4c818e603: Layer already exists 1698c1b7e3e6: Layer already exists 227442bb48dc: Layer already exists d899691659b0: Pushing [==================================================>] 64.16 MB95ef25a32043: Pushing [==================================================>] 72.49 MBerror parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.14.1</center>\r\n</body>\r\n</html>\r\n"
Quay Image: quay.io/projectquay/quay:50ba6130
Steps:
- Deploy Quay with backend storage as Azure Blob Storage
- Create new image repository
- Push image to new image repository
Expected Results:
Push image should complete successfully.
Actual Results:
Push image was failed.
Quay POD Logs:
nginx stdout | 2020/08/07 07:24:59 [error] 113#0: *105131 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.128.2.3, server: _, request: "PATCH /v2/qateam/demo/blobs/uploads/15b7c748-6b4c-4c0a-a770-9667172ac0b9 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/qateam/demo/blobs/uploads/15b7c748-6b4c-4c0a-a770-9667172ac0b9", host: "demo-quayecosystem-quay-quay-enterprise.apps.quay-286.qe.azure.devcluster.openshift.com" nginx stdout | 2020/08/07 07:24:59 [error] 113#0: *105131 client intended to send too large body: 19933288 bytes, client: 10.128.2.3, server: _, request: "PATCH /v2/qateam/demo/blobs/uploads/15b7c748-6b4c-4c0a-a770-9667172ac0b9 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock/v2/qateam/demo/blobs/uploads/15b7c748-6b4c-4c0a-a770-9667172ac0b9", host: "demo-quayecosystem-quay-quay-enterprise.apps.quay-286.qe.azure.devcluster.openshift.com" nginx stdout | 2020/08/07 07:24:59 [error] 113#0: *105129 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.128.2.3, server: _, request: "PATCH /v2/qateam/demo/blobs/uploads/cb42dd0e-4a08-41ad-85d1-d79f7d2608b5 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/qateam/demo/blobs/uploads/cb42dd0e-4a08-41ad-85d1-d79f7d2608b5", host: "demo-quayecosystem-quay-quay-enterprise.apps.quay-286.qe.azure.devcluster.openshift.com" nginx stdout | 2020/08/07 07:24:59 [error] 113#0: *105129 client intended to send too large body: 20149981 bytes, client: 10.128.2.3, server: _, request: "PATCH /v2/qateam/demo/blobs/uploads/cb42dd0e-4a08-41ad-85d1-d79f7d2608b5 HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock/v2/qateam/demo/blobs/uploads/cb42dd0e-4a08-41ad-85d1-d79f7d2608b5", host: "demo-quayecosystem-quay-quay-enterprise.apps.quay-286.qe.azure.devcluster.openshift.com"
