README:
--------
Quay version is v3.10.6. Our Quay install runs on standalone EC2 instances, Storage is AWS S3 and the DB is AWS RDS Aurora PostGres. We dont run Quay on OpenShift.

Issue:

---

We're seeing intermittent issue with puling images through a Org configured as a proxy cache, where when pulling images a 400 is returned to the docker client pulling the image:

"Dec 04 06:39:27 ddivilly-proxy-cache dockerd[325421]: time="2024-12-04T06:39:27.164584658-08:00" level=info msg="Attempting next endpoint for pull after error: error pulling image configuration: download failed after attempts=1: error parsing HTTP 400 response body: no error details found in HTTP response body: \"

{\\\"message\\\":\\\"('No storage found with uuid: %s', '7dfbdbf1-f614-4780-8b91-9cb6178a2445')\\\"}

\\n\"" spanID=3bfb50443e1eadeb traceID=00aceeb27365e945c14c626a098270f0"

Checking the Quay debug logs we see these entries which correspond to what is returned to the client.

"{"log":"gunicorn-registry stdout | 2024-12-04 14:39:27,137 [293] [DEBUG] [app] Ending request: urn:request:6047241e-9cf9-4dc8-b4e0-ae2ee197967f (/v2/proxy-cache-qcom/library/golang/blobs/sha256:1983e5a25166e38fd5cdd05e7c43abe3764ed1e2dd27746667d3e7fc0e4fb07f) {'endpoint': 'v2.download_blob', 'request_id': 'urn:request:6047241e-9cf9-4dc8-b4e0-ae2ee197967f', 'remote_addr': '10.81.37.78', 'http_method': 'GET', 'original_url': 'https://proxy-cache.qualcomm.com/v2/proxy-cache-qcom/library/golang/blobs/sha256:1983e5a25166e38fd5cdd05e7c43abe3764ed1e2dd27746667d3e7fc0e4fb07f', 'path': '/v2/proxy-cache-qcom/library/golang/blobs/sha256:1983e5a25166e38fd5cdd05e7c43abe3764ed1e2dd27746667d3e7fc0e4fb07f', 'parameters': {}, 'json_body': None, 'confsha': '4b60b93e', 'user-agent': 'docker/27.3.1 go/go1.22.7 git-commit/41ca978 kernel/5.15.0-126-generic os/linux arch/amd64 UpstreamClient(Docker-Client/27.3.1 \\\\(linux\\\\))'}\n","stream":"stdout","time":"2024-12-04T14:39:27.137755278Z"}
Show syntax highlighted

{"log":"gunicorn-registry stdout | data.model.InvalidImageException: ('No storage found with uuid: %s', '7dfbdbf1-f614-4780-8b91-9cb6178a2445')\n","stream":"stdout","time":"2024-12-04T14:39:27.134196944Z"}

Show syntax highlighted

{"log":"gunicorn-registry stdout | raise InvalidImageException(\"No storage found with uuid: %s\", storage_uuid)\n","stream":"stdout","time":"2024-12-04T14:39:27.134195264Z"}

Show syntax highlighted

{"log":"gunicorn-registry stdout | File \"/quay-registry/data/model/storage.py\", line 298, in get_storage_by_uuid\n","stream":"stdout","time":"2024-12-04T14:39:27.134186823Z"}

Show syntax highlighted

{"log":"gunicorn-registry stdout | return get_storage_by_uuid(storage.uuid) if storage is not None else None\n","stream":"stdout","time":"2024-12-04T14:39:27.134185513Z"}

"

When we try to pull the same image/tag (docker pull golang:latest) a second time it works.