During an upgrade from 3.9.x to 3.10.x, the upgrade can fail with the following error message:

Entering migration mode to version: head
15:59:19 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
15:59:19 INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
15:59:19 INFO  [alembic.runtime.migration] Running upgrade b82361fba1cd -> 4366f54be43c, add indexes to quotanamespacesize and quotarepositorysize tables
Traceback (most recent call last):
  File "/app/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
    self.dialect.do_execute(
  File "/app/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UniqueViolation: could not create unique index "quotanamespacesize_namespace_user_id"
DETAIL:  Key (namespace_user_id)=(10100) is duplicated.

The alembic migration noted here creates indexes on the quotarepositorysize and quotanamespacesize tables and creation of indexes fail due to duplicate values in the table. We suspect that the cause is a race condition between two workers when they try to calculate the size for the same namespace. Because there is no check on whether the repo is being used and in 3.9 there's no constraint on the namespace_user_id column in the quotanamespacesize table, each worker creates an entry in the table which then causes the index creation failure.

Current workaround is to wipe both tables, when these are large, identifying duplicate entires is very cumbersome, if not impossible, we also don't know which value is correct of the two values that are stoed. Unfortunately, wiping these tables requires Quay to backfill sizes from scratch, for big registries this might pose a problem.

Please check!