Details
-
Bug
-
Resolution: Done-Errata
-
Critical
-
None
Description
We often see garbage collection stopping due to too large values in the removed_tag_expiration_s column of the user table. This column is currently set to integer which is fine for most uses, but becomes problematic with garbage collection since we multiply this value by 1000. The maximum allowed number for a signed integer is [(1/2)*2^32]-1 or 2147483647, any number over this value will produce the following Python trace:
gcworker stdout | 2021-08-16 08:19:40,562 [106] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."repository_id", "t1"."manifest_id", "t1"."lifetime_start_ms", "t1"."lifetime_end_ms", "t1"."hidden", "t1"."reversion", "t1"."tag_kind_id", "t1"."linked_tag_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON (" t2"."namespace_user_id" = "t3"."id") WHERE ((("t1"."repository_id" = %s) AND NOT ("t1"."lifetime_end_ms" IS %s)) AND ("t1"."lifetime_end_ms" <= (%s - ("t3"."removed_tag_expiration_s" * %s)))) LIMIT %s', [404, None, 1629101980561, 1000, 10]) gcworker stdout | 2021-08-16 08:19:40,565 [106] [DEBUG] [data.database] Disconnecting from database. 2021-08-16 08:19:40,565 [106] [ERROR] [workers.worker] Operation raised exception Traceback (most recent call last): File "workers/worker.py", line 87, in _operation_func return operation_func() File "/quay-registry/workers/gc/gcworker.py", line 47, in _garbage_collection_repos garbage_collect_repo(repository) File "data/model/gc.py", line 247, in garbage_collect_repo for tags in _chunk_iterate_for_deletion(oci_tag.lookup_unrecoverable_tags(repo)): File "data/model/gc.py", line 164, in _chunk_iterate_for_deletion results = list(query.limit(chunk_size)) File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 6655, in __iter__ self.execute() File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 1812, in inner return method(self, database, *args, **kwargs) File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 1883, in execute return self._execute(database) File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2054, in _execute cursor = database.execute(self) File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 3018, in execute return self.execute_sql(sql, params, commit=commit) File "data/database.py", line 210, in execute_sql cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit) File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 3012, in execute_sql self.commit() File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2783, in __exit__ reraise(new_type, new_type(*exc_args), traceback) File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 3005, in execute_sql cursor.execute(sql, params or ()) DataError: integer out of range
For this user, the removed_tag_expiration_s was set to 2419200 or 4 weeks worth of seconds, which clearly overflows the query.
The issue affects all versions of Quay.
