Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-3976

Pull-thru gives 500 when pulling certain images

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • quay-v3.7.7, quay-v3.8.0
    • quay-v3.7.1, quay-v3.7.2, quay-v3.7.3, quay-v3.7.4, quay-v3.7.5
    • quay
    • 0

    Description

      Deleting the image (or repository, uncertain) seems to resolve the problem, but it's challenging to do because if there's a pull right after deletion Quay will restore the image.

      Problematic images, as reported by customer:

      • registry.redhat.io/redhat/redhat-operator-index:v4.10
      • registry.redhat.io/openshift3/ose-cli

      Steps to reproduce (adjust with your namespace and quay instance host:port)

      • skopeo copy --all --remove-signatures --dest-tls-verify=false docker://library/nginx:1.23.1 docker://quay.io/fmissi/nginx:1.23.1
      • skopeo copy --all --remove-signatures --dest-tls-verify=false docker://library/nginx:1.23.1 docker://quay.io/fmissi/nginx:latest
      • podman pull quay:8080/cache-quayio/fmissi/nginx:latest # this pull will succeed
      • skopeo copy --all --remove-signatures --dest-tls-verify=false docker://library/nginx:1.22 docker://quay.io/fmissi/nginx:1.22
      • skopeo copy --all --remove-signatures --dest-tls-verify=false docker://library/nginx:1.22 docker://quay.io/fmissi/nginx:latest
      • podman pull quay:8080/cache-quayio/fmissi/nginx:latest # this will also succeed
      • skopeo copy --all --remove-signatures --dest-tls-verify=false docker://library/nginx:1.23.1 docker://quay.io/fmissi/nginx:latest
      • podman pull quay:8080/cache-quayio/fmissi/nginx:latest # this will fail!

      Logs

      gunicorn-registry stdout | 2022-06-10 11:25:39,421 [254] [ERROR] [gunicorn.error] Error handling request /v2/redhatio/openshift3/ose-cli/manifests/sha256:193d7245f86083a268409c58fc88baed7148b65475b51ce7ca45c9d40ad7b979
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 3057, in execute_sql
gunicorn-registry stdout |     cursor.execute(sql, params or ())
gunicorn-registry stdout | psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "tag_repository_id_name_lifetime_end_ms"
gunicorn-registry stdout | DETAIL:  Key (repository_id, name, lifetime_end_ms)=(85, latest, 1662636339410) already exists.
gunicorn-registry stdout | During handling of the above exception, another exception occurred:
gunicorn-registry stdout | Traceback (most recent call last):
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle
gunicorn-registry stdout |     self.handle_request(listener_name, req, client, addr)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 127, in handle_request
gunicorn-registry stdout |     super().handle_request(listener_name, req, sock, addr)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request
gunicorn-registry stdout |     respiter = self.wsgi(environ, resp.start_response)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 2463, in __call__
gunicorn-registry stdout |     return self.wsgi_app(environ, start_response)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
gunicorn-registry stdout |     return self.app(environ, start_response)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app
gunicorn-registry stdout |     response = self.handle_exception(e)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1866, in handle_exception
gunicorn-registry stdout |     reraise(exc_type, exc_value, tb)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
gunicorn-registry stdout |     raise value
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app
gunicorn-registry stdout |     response = self.full_dispatch_request()
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request
gunicorn-registry stdout |     rv = self.handle_user_exception(e)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception
gunicorn-registry stdout |     reraise(exc_type, exc_value, tb)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
gunicorn-registry stdout |     raise value
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request
gunicorn-registry stdout |     rv = self.dispatch_request()
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request
gunicorn-registry stdout |     return self.view_functions[rule.endpoint](**req.view_args)
gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 218, in wrapper
gunicorn-registry stdout |     return func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 91, in wrapper
gunicorn-registry stdout |     return func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/auth/registry_jwt_auth.py", line 177, in wrapper
gunicorn-registry stdout |     return func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/__init__.py", line 143, in wrapped
gunicorn-registry stdout |     return func(namespace_name, repo_name, *args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 154, in wrapper
gunicorn-registry stdout |     return func(*args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/endpoints/decorators.py", line 54, in wrapper
gunicorn-registry stdout |     return func(namespace_name, repo_name, *args, **kwargs)
gunicorn-registry stdout |   File "/quay-registry/endpoints/v2/manifest.py", line 144, in fetch_manifest_by_digest
gunicorn-registry stdout |     manifest = registry_model.lookup_manifest_by_digest(
gunicorn-registry stdout |   File "/quay-registry/data/registry_model/registry_proxy_model.py", line 191, in lookup_manifest_by_digest
gunicorn-registry stdout |     TagTable.update(lifetime_end_ms=new_expiration).where(
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 1845, in inner
gunicorn-registry stdout |     return method(self, database, *args, **kwargs)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 1916, in execute
gunicorn-registry stdout |     return self._execute(database)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 2400, in _execute
gunicorn-registry stdout |     cursor = database.execute(self)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 3070, in execute
gunicorn-registry stdout |     return self.execute_sql(sql, params, commit=commit)
gunicorn-registry stdout |   File "/quay-registry/data/database.py", line 226, in execute_sql
gunicorn-registry stdout |     cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 3064, in execute_sql
gunicorn-registry stdout |     self.commit()
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 2831, in __exit__
gunicorn-registry stdout |     reraise(new_type, new_type(exc_value, *exc_args), traceback)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 183, in reraise
gunicorn-registry stdout |     raise value.with_traceback(tb)
gunicorn-registry stdout |   File "/app/lib/python3.8/site-packages/peewee.py", line 3057, in execute_sql
gunicorn-registry stdout |     cursor.execute(sql, params or ())
gunicorn-registry stdout | peewee.IntegrityError: duplicate key value violates unique constraint "tag_repository_id_name_lifetime_end_ms"
gunicorn-registry stdout | DETAIL:  Key (repository_id, name, lifetime_end_ms)=(85, latest, 1662636339410) already exists.

      Attachments

        Activity

          People

            fmissi Flavian Missi
            fmissi Flavian Missi
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: