-
Bug
-
Resolution: Obsolete
-
Major
-
quay-v3.2.0
The error on the client (docker push) looks like this:
error parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"
The error is accompanied by following messages in quay logs:
nginx stdout | 2020/01/07 10:55:25 [alert] 222#0: *263407 writev() failed (105: No buffer space available) while sending request to upstream, client: 10.12.227.241, server: , request: "PATCH /v2/some_namespace/some_image/blobs/uploads/d2a32855-9528-440e-81d4-98454553d88d HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_registry.sock:/v2/some_namespace/some_image/blobs/uploads/d2a32855-9528-440e-81d4-98454553d88d", host: "quay-preview.domain.tld"
nginx stdout | 10.12.227.241 () - - [07/Jan/2020:10:55:25 +0000] "PATCH /v2/some_namespace/some_image/blobs/uploads/d2a32855-9528-440e-81d4-98454553d88d HTTP/1.1" 413 199 "-" "docker/19.03.5 go/go1.12.12 git-commit/633a0ea838 kernel/4.15.0-50-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.5 \x5C(linux\x5C))" (1.083 3611001 1.035)
2020-01-07 10:55:25,745 [651] [WARNING] [storage.cloud] Error when writing to stream in stream_write_internal at path uploads/fe6e93a3-b879-4b39-97bf-255b36e64e6e: No more data after: None
gunicorn-registry stdout | 2020-01-07 10:55:25,745 [651] [WARNING] [storage.cloud] Error when writing to stream in stream_write_internal at path uploads/fe6e93a3-b879-4b39-97bf-255b36e64e6e: No more data after: None
2020-01-07 10:55:25,746 [651] [ERROR] [data.registry_model.blobuploader] storage.stream_upload_chunk returned error No more data after: None
gunicorn-registry stdout | 2020-01-07 10:55:25,746 [651] [ERROR] [data.registry_model.blobuploader] storage.stream_upload_chunk returned error No more data after: None
2020-01-07 10:55:25,747 [651] [ERROR] [endpoints.v2.blob] Exception when uploading blob to d2a32855-9528-440e-81d4-98454553d88d
Traceback (most recent call last):
File "/quay-registry/endpoints/v2/blob.py", line 427, in _upload_chunk
blob_uploader.upload_chunk(app.config, input_fp, start_offset, length, metric_queue)
File "/quay-registry/data/registry_model/blobuploader.py", line 204, in upload_chunk
raise BlobUploadException(upload_error)
BlobUploadException: No more data after: None
We are using the RadosGWStorage with an s3/radosgw compatible appliance:
DISTRIBUTED_STORAGE_CONFIG:
default:
- RadosGWStorage
- {access_key: quay-dev, bucket_name: dev, hostname: somehost.domain.tld, is_secure: true,
port: '443', secret_key: '<snip>', storage_path: /datastore}
I would expect that docker push for the same image would either consistently fail or consistently succeed. I would expect that pushing small layers would succeed without running into 413 Request Entity Too Large.
The same setup worked with quay 2.9.5.
Tested with 3.2.0, still broken
