Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-3818

pull-thru gives 500 when manifest list's sub-manifest is already proxied under different tag in same repo

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • quay-v3.7.1
    • quay-v3.7.0
    • quay
    • 0

    Description

      Reported by EAP customer via email.

      Logs below.

      Steps to reproduce

      1. Setup a proxy org to proxy docker.io/library
      2. pull python:bullseye thru the proxy org
      3. pull python:latest thru the proxy org

      With podman the steps above yield:

      Error: determining manifest MIME type for docker://quay:8080/cache/python:latest: reading manifest sha256:ec43d739179d1979274d05ac081e279c97cfe5ca31777b7de3ec77ff82909073 in quay:8080/cache/python: received unexpected HTTP status: 500 Internal Server Error

      Logs
      Note that the query in the first line of the logs is the culprit for the error. It's a bit confusing because the query object is given to the next query and that's where the exception occurs. The code is here: https://github.com/quay/quay/blob/e75397ddfe18815a320a926e6aad06ef28df1310/data/registry_model/registry_proxy_model.py?plain=1#L185-L188

      quay-quay  | gunicorn-registry stdout | 2022-05-18 15:46:10,038 [249] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."repository_id", "t1"."manifest_id", "t1"."lifetime_start_ms", "t1"."lifetime_end_ms", "t1"."hidden", "t1"."reversion", "t1"."tag_kind_id", "t1"."linked_tag_id", "t2"."id", "t2"."repository_id", "t2"."digest", "t2"."media_type_id", "t2"."manifest_bytes", "t2"."config_media_type", "t2"."layers_compressed_size" FROM "tag" AS "t1" INNER JOIN "manifest" AS "t2" ON ("t1"."manifest_id" = "t2"."id") WHERE (("t1"."repository_id" = %s) AND ("t1"."manifest_id" = (SELECT "t3"."manifest_id" FROM "manifestchild" AS "t3" WHERE (("t3"."repository_id" = %s) AND ("t3"."child_manifest_id" = %s))))) ORDER BY "t1"."lifetime_end_ms" DESC LIMIT %s OFFSET %s', [26, 26, 88, 1, 0])
quay-quay  | gunicorn-registry stdout | 2022-05-18 15:46:10,041 [249] [DEBUG] [data.database] Disconnecting from database.
quay-quay  | gunicorn-registry stdout | 2022-05-18 15:46:10,041 [249] [DEBUG] [peewee.pool] Returning 140360620538608 to pool.
quay-quay  | gunicorn-registry stdout | 2022-05-18 15:46:10,041 [249] [ERROR] [gunicorn.error] Error handling request /v2/cache/python/manifests/sha256:ec43d739179d1979274d05ac081e279c97cfe5ca31777b7de3ec77ff82909073
quay-quay  | gunicorn-registry stdout | Traceback (most recent call last):
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 3057, in execute_sql
quay-quay  | gunicorn-registry stdout |     cursor.execute(sql, params or ())
quay-quay  | gunicorn-registry stdout | psycopg2.errors.CardinalityViolation: more than one row returned by a subquery used as an expression
quay-quay  | gunicorn-registry stdout | During handling of the above exception, another exception occurred:
quay-quay  | gunicorn-registry stdout | Traceback (most recent call last):
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
quay-quay  | gunicorn-registry stdout |     self.handle_request(listener_name, req, client, addr)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 127, in handle_request
quay-quay  | gunicorn-registry stdout |     super().handle_request(listener_name, req, sock, addr)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request
quay-quay  | gunicorn-registry stdout |     respiter = self.wsgi(environ, resp.start_response)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 2463, in __call__
quay-quay  | gunicorn-registry stdout |     return self.wsgi_app(environ, start_response)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
quay-quay  | gunicorn-registry stdout |     return self.app(environ, start_response)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app
quay-quay  | gunicorn-registry stdout |     response = self.handle_exception(e)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1866, in handle_exception
quay-quay  | gunicorn-registry stdout |     reraise(exc_type, exc_value, tb)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
quay-quay  | gunicorn-registry stdout |     raise value
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
quay-quay  | gunicorn-registry stdout |     response = self.full_dispatch_request()
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request
quay-quay  | gunicorn-registry stdout |     rv = self.handle_user_exception(e)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
quay-quay  | gunicorn-registry stdout |     reraise(exc_type, exc_value, tb)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
quay-quay  | gunicorn-registry stdout |     raise value
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
quay-quay  | gunicorn-registry stdout |     rv = self.dispatch_request()
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
quay-quay  | gunicorn-registry stdout |     return self.view_functions[rule.endpoint](**req.view_args)
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 218, in wrapper
quay-quay  | gunicorn-registry stdout |     return func(*args, **kwargs)
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 91, in wrapper
quay-quay  | gunicorn-registry stdout |     return func(*args, **kwargs)
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/auth/registry_jwt_auth.py", line 177, in wrapper
quay-quay  | gunicorn-registry stdout |     return func(*args, **kwargs)
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/__init__.py", line 143, in wrapped
quay-quay  | gunicorn-registry stdout |     return func(namespace_name, repo_name, *args, **kwargs)
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 154, in wrapper
quay-quay  | gunicorn-registry stdout |     return func(*args, **kwargs)
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 54, in wrapper
quay-quay  | gunicorn-registry stdout |     return func(namespace_name, repo_name, *args, **kwargs)
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/manifest.py", line 144, in fetch_manifest_by_digest
quay-quay  | gunicorn-registry stdout |     manifest = registry_model.lookup_manifest_by_digest(
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/data/registry_model/registry_proxy_model.py", line 189, in lookup_manifest_by_digest
quay-quay  | gunicorn-registry stdout |     parent_tag = oci.tag.get_tag_by_manifest_id(repository_ref.id, parent)
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/data/model/oci/tag.py", line 60, in get_tag_by_manifest_id
quay-quay  | gunicorn-registry stdout |     Tag.select(Tag, Manifest)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 6741, in get
quay-quay  | gunicorn-registry stdout |     return clone.execute(database)[0]
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 1845, in inner
quay-quay  | gunicorn-registry stdout |     return method(self, database, *args, **kwargs)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 1916, in execute
quay-quay  | gunicorn-registry stdout |     return self._execute(database)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 2087, in _execute
quay-quay  | gunicorn-registry stdout |     cursor = database.execute(self)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 3070, in execute
quay-quay  | gunicorn-registry stdout |     return self.execute_sql(sql, params, commit=commit)
quay-quay  | gunicorn-registry stdout |   File "/quay-registry/data/database.py", line 226, in execute_sql
quay-quay  | gunicorn-registry stdout |     cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 3064, in execute_sql
quay-quay  | gunicorn-registry stdout |     self.commit()
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 2831, in __exit__
quay-quay  | gunicorn-registry stdout |     reraise(new_type, new_type(exc_value, *exc_args), traceback)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 183, in reraise
quay-quay  | gunicorn-registry stdout |     raise value.with_traceback(tb)
quay-quay  | gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 3057, in execute_sql
quay-quay  | gunicorn-registry stdout |     cursor.execute(sql, params or ())
quay-quay  | gunicorn-registry stdout | peewee.ProgrammingError: more than one row returned by a subquery used as an expression

      Attachments

        Issue Links

          causes

          Bug - A problem that impairs or prevents the functions of the product. PROJQUAY-3976 Pull-thru gives 500 when pulling certain images

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          links to

          GitHub quay/quay#1336: pull-thru: bump expiration of all parent manifest lists on pull (PROJQUAY-3818)

          Activity

            People

              fmissi Flavian Missi
              fmissi Flavian Missi
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: