Details
-
Bug
-
Resolution: Done-Errata
-
Critical
-
quay-v3.7.11, quay-v3.8.0, quay-v3.9.5
Description
I found a rather specific case with a proxy cache.
We use Quay with Proxy Cache feature and k8s-image-availability-exporter for Prometheus to check availability of docker images for our services in Kubernetes.
This exporter check image availability with HTTP Head method.
I noticed that after caching docker image then manifest is a list child manifests tag lifetime end is not updated.
Main manifest list tag is updated because prometheus exporter request docker image tag. As I wrote above it doesn`t pull docker image by tag, but uses HTTP Head method instead. Therefore, there is no subsequent access to the child manifests tag.
After child manifest tag lifetime end GC remove record from tag table in DB and we got exception then try to pull docker image by tag.
gunicorn-registry stdout | 2022-12-21 10:51:52,417 [215] [ERROR] [gunicorn.error] Error handling request /v2/proxy/appscode/stash/manifests/sha256:9c0886ba503096ce99b9e60191e7df099835fb388bacc2b5bb3704465c103884 gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 55, in handle gunicorn-registry stdout | self.handle_request(listener_name, req, client, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/ggevent.py", line 127, in handle_request gunicorn-registry stdout | super().handle_request(listener_name, req, sock, addr) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request gunicorn-registry stdout | respiter = self.wsgi(environ, resp.start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2463, in __call__ gunicorn-registry stdout | return self.wsgi_app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__ gunicorn-registry stdout | return self.app(environ, start_response) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2449, in wsgi_app gunicorn-registry stdout | response = self.handle_exception(e) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1866, in handle_exception gunicorn-registry stdout | reraise(exc_type, exc_value, tb) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise gunicorn-registry stdout | raise value gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 2446, in wsgi_app gunicorn-registry stdout | response = self.full_dispatch_request() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1951, in full_dispatch_request gunicorn-registry stdout | rv = self.handle_user_exception(e) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1820, in handle_user_exception gunicorn-registry stdout | reraise(exc_type, exc_value, tb) gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise gunicorn-registry stdout | raise value gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1949, in full_dispatch_request gunicorn-registry stdout | rv = self.dispatch_request() gunicorn-registry stdout | File "/app/lib/python3.9/site-packages/flask/app.py", line 1935, in dispatch_request gunicorn-registry stdout | return self.view_functions[rule.endpoint](**req.view_args) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 228, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 91, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/auth/registry_jwt_auth.py", line 177, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/__init__.py", line 200, in wrapped gunicorn-registry stdout | return func(namespace_name, repo_name, *args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 164, in wrapper gunicorn-registry stdout | return func(*args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/decorators.py", line 54, in wrapper gunicorn-registry stdout | return func(namespace_name, repo_name, *args, **kwargs) gunicorn-registry stdout | File "/quay-registry/endpoints/v2/manifest.py", line 144, in fetch_manifest_by_digest gunicorn-registry stdout | manifest = registry_model.lookup_manifest_by_digest( gunicorn-registry stdout | File "/quay-registry/data/registry_model/registry_proxy_model.py", line 208, in lookup_manifest_by_digest gunicorn-registry stdout | f.write(repository_ref.id) gunicorn-registry stdout | AttributeError: 'NoneType' object has no attribute 'manifest' gunicorn-registry stdout | 2022-12-21 10:51:52,418 [215] [INFO] [gunicorn.access] - - [21/Dec/2022:10:51:52 +0000] "GET /v2/proxy/appscode/stash/manifests/sha256:9c0886ba503096ce99b9e60191e7df099835fb388bacc2b5bb3704465c103884 HTTP/1.1" 500 0 "-" "-"nginx stdout | 10.131.113.196 (-) - - [21/Dec/2022:10:51:52 +0000] "GET /v2/proxy/appscode/stash/manifests/sha256:9c0886ba503096ce99b9e60191e7df099835fb388bacc2b5bb3704465c103884 HTTP/1.1" 500 141 "-" "curl/7.85.0" (0.033 457 0.032)
This exception occurs because of this code. Child manifest exist in manifest table in DB but record for its temporary tag not.
To download docker image again via proxy, you need to delete the repository. Deleting the tag doesn't help.
As a solution, we can update child manifests tags lifetime end when requesting parent manifest or the main tag. By analogy with this code for updating main tag lifetime end then we requesting child manifest.
I can do PR with this code change.
Attachments
Issue Links
- links to
-
RHBA-2024:126633
Red Hat Quay v3.10.3 minor release
- mentioned on
