-
Bug
-
Resolution: Done
-
Blocker
-
quay-v3.7.0
Description:
This is an issue found when push image to Quay 3.7.0, after push multiple images to Quay image repo, sometimes hit internal 500 error, checked Quay POD. get error "peewee.IntegrityError: duplicate key value violates unique constraint "repositorysize_repository_id", but here Quay Postgres DB is managed by Quay Operator, pls review this issue, attached Quay APP POD logs quay370_app_pod2.logs
Quay Image: quay-operator-bundle-container-v3.7.0-112
Note: When hit this kind of issue, retry to push will be passed.
[root@ip-10-0-1-148 fedora]# podman push quay370.apps.quayperf370.perfscale.devcluster.openshift.com/larryteam/gitlab --tls-verify=false Getting image source signatures Copying blob 3a78b4f27fdd done Copying blob b52ac289fc65 [--------------------------------------] 8.0b / 5.5KiB Copying blob 3fb324f8944b done Copying blob bf579549b483 done Copying blob 04ed5367b77f done Copying blob bf8cedc62fb3 done Copying blob 4acb1b593bd4 done Copying blob 3aec0ed09dd8 done Copying blob 82bffbeafaec done Copying blob a26ceb4ed710 done Error: writing blob: initiating layer upload to /v2/larryteam/gitlab/blobs/uploads/ in quay370.apps.quayperf370.perfscale.devcluster.openshift.com: received unexpected HTTP status: 500 Internal Server Error Retry to push will be completed. [root@ip-10-0-1-148 fedora]# podman push quay370.apps.quayperf370.perfscale.devcluster.openshift.com/larryteam/gitlab --tls-verify=false Getting image source signatures Copying blob b52ac289fc65 done Copying blob bf8cedc62fb3 skipped: already exists Copying blob 3fb324f8944b skipped: already exists Copying blob bf579549b483 skipped: already exists Copying blob 04ed5367b77f skipped: already exists Copying blob 3a78b4f27fdd skipped: already exists Copying blob 4acb1b593bd4 skipped: already exists Copying blob a26ceb4ed710 skipped: already exists Copying blob 82bffbeafaec skipped: already exists Copying blob 3aec0ed09dd8 skipped: already exists Copying config 27dd1be4be done Writing manifest to image destination Storing signatures
Quay APP POD logs:
gunicorn-registry stdout | 2022-05-05 07:36:51,577 [227] [ERROR] [gunicorn.error] Error handling request /v2/larryteam/gitlab/blobs/uploads/ gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3057, in execute_sql gunicorn-registry stdout | cursor.execute(sql, params or ()) gunicorn-registry stdout | psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "repositorysize_repository_id" gunicorn-registry stdout | DETAIL: Key (repository_id)=(6) already exists. gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 55, in handle gunicorn-registry stdout | self.handle_request(listener_name, req, client, addr) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/ggevent.py", line 127, in handle_request gunicorn-registry stdout | super().handle_request(listener_name, req, sock, addr) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base_async.py", line 108, in handle_request gunicorn-registry stdout | respiter = self.wsgi(environ, resp.start_response) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2463, in __call__ gunicorn-registry stdout | return self.wsgi_app(environ, start_response) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__ gunicorn-registry stdout | return self.app(environ, start_response) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2449, in wsgi_app gunicorn-registry stdout | response = self.handle_exception(e) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1866, in handle_exception gunicorn-registry stdout | reraise(exc_type, exc_value, tb) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise gunicorn-registry stdout | raise value gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2446, in wsgi_app gunicorn-registry stdout | response = self.full_dispatch_request() gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1951, in full_dispatch_request gunicorn-registry stdout | rv = self.handle_user_exception(e) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in handle_user_exception gunicorn-registry stdout | reraise(exc_type, exc_value, tb) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise gunicorn-registry stdout | raise value gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1949, in full_dispatch_request gunicorn-registry stdout | rv = self.dispatch_request() gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1935, in dispatch_request gunicorn-registry stdout | return self.view_functions[rule.endpoint](**req.view_args) gunicorn-registry stdout | File "/quay-registry/data/model/repository.py", line 697, in get_repository_size_and_cache gunicorn-registry stdout | force_cache_repo_size(repo_id) gunicorn-registry stdout | File "/quay-registry/data/model/repository.py", line 753, in force_cache_repo_size gunicorn-registry stdout | RepositorySize.create(repository_id=repo_id, size_bytes=size) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 6235, in create gunicorn-registry stdout | inst.save(force_insert=True) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 6433, in save gunicorn-registry stdout | pk = self.insert(**field_dict).execute() gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 1845, in inner gunicorn-registry stdout | return method(self, database, *args, **kwargs) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 1916, in execute gunicorn-registry stdout | return self._execute(database) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2665, in _execute gunicorn-registry stdout | return super(Insert, self)._execute(database) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2398, in _execute gunicorn-registry stdout | cursor = self.execute_returning(database) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2405, in execute_returning gunicorn-registry stdout | cursor = database.execute(self) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3070, in execute gunicorn-registry stdout | return self.execute_sql(sql, params, commit=commit) gunicorn-registry stdout | File "/quay-registry/data/database.py", line 226, in execute_sql gunicorn-registry stdout | cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3064, in execute_sql gunicorn-registry stdout | self.commit() gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 2831, in __exit__ gunicorn-registry stdout | reraise(new_type, new_type(exc_value, *exc_args), traceback) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 183, in reraise gunicorn-registry stdout | raise value.with_traceback(tb) gunicorn-registry stdout | File "/usr/local/lib/python3.8/site-packages/peewee.py", line 3057, in execute_sql gunicorn-registry stdout | cursor.execute(sql, params or ()) gunicorn-registry stdout | peewee.IntegrityError: duplicate key value violates unique constraint "repositorysize_repository_id" gunicorn-registry stdout | DETAIL: Key (repository_id)=(6) already exists. gunicorn-registry stdout | 2022-05-05 07:36:51,581 [227] [INFO] [gunicorn.access] - - [05/May/2022:07:36:51 +0000] "POST /v2/larryteam/gitlab/blobs/uploads/ HTTP/1.1" 500 0 "-" "-" nginx stdout | 10.128.2.12 () - - [05/May/2022:07:36:51 +0000] "POST /v2/larryteam/gitlab/blobs/uploads/ HTTP/1.1" 500 141 "-" "containers/5.15.2 (github.com/containers/image)" (0.031 1363 0.032) gunicorn-registry stdout | 2022-05-05 07:36:51,583 [220] [DEBUG] [peewee] ('SELECT sum("t1"."layers_compressed_size") FROM "manifest" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN (SELECT "t3"."repository_id" FROM "tag" AS "t3" WHERE (("t3"."hidden" = %s) AND (("t3"."lifetime_end_ms" IS %s) OR ("t3"."lifetime_end_ms" > %s))) GROUP BY "t3"."repository_id" HAVING (Count("t3"."name") > %s)) AS "t4" ON ("t4"."repository_id" = "t2"."id") WHERE ("t2"."namespace_user_id" = %s)', [False, None, 1651736211582, 0, 6])
- is related to
-
-
- Closed
-
