-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
omr-v2.0.5
-
Incidents & Support
-
5
-
False
-
-
False
-
Quay Enterprise
-
-
When Quay is trying to do incompatible operations on the SQLite database on the same tables at the same time (for instance, SELECT/DELETE), the database can end up being locked. This is shown in the logs as a Peewee exception such as this one:
gcworker stdout | sqlite3.OperationalError: database is locked gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/data/database.py", line 228, in execute_sql gcworker stdout | cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit) gcworker stdout | File "/app/lib/python3.9/site-packages/peewee.py", line 3064, in execute_sql gcworker stdout | self.commit() gcworker stdout | File "/app/lib/python3.9/site-packages/peewee.py", line 2831, in __exit__ gcworker stdout | reraise(new_type, new_type(exc_value, *exc_args), traceback) gcworker stdout | File "/app/lib/python3.9/site-packages/peewee.py", line 183, in reraise gcworker stdout | raise value.with_traceback(tb) gcworker stdout | File "/app/lib/python3.9/site-packages/peewee.py", line 3057, in execute_sql gcworker stdout | cursor.execute(sql, params or ()) gcworker stdout | peewee.OperationalError: database is locked gcworker stdout | During handling of the above exception, another exception occurred: gcworker stdout | Traceback (most recent call last): gcworker stdout | File "/quay-registry/workers/worker.py", line 86, in _operation_func gcworker stdout | return operation_func() gcworker stdout | File "/quay-registry/workers/gc/gcworker.py", line 77, in _garbage_collection_repos gcworker stdout | garbage_collect_repo(repository) gcworker stdout | File "/quay-registry/data/model/gc.py", line 248, in garbage_collect_repo gcworker stdout | _purge_oci_tag(tag, context) gcworker stdout | File "/quay-registry/data/model/gc.py", line 319, in _purge_oci_tag gcworker stdout | delete_tag_notifications_for_tag(tag) gcworker stdout | File "/quay-registry/data/model/notification.py", line 287, in delete_tag_notifications_for_tag gcworker stdout | resp = TagNotificationSuccess.delete().where(TagNotificationSuccess.tag == tag.id).execute() gcworker stdout | File "/app/lib/python3.9/site-packages/peewee.py", line 1845, in inner gcworker stdout | return method(self, database, *args, **kwargs) gcworker stdout | File "/app/lib/python3.9/site-packages/peewee.py", line 1916, in execute gcworker stdout | return self._execute(database) gcworker stdout | File "/app/lib/python3.9/site-packages/peewee.py", line 2400, in _execute gcworker stdout | cursor = database.execute(self) gcworker stdout | File "/app/lib/python3.9/site-packages/peewee.py", line 3070, in execute gcworker stdout | return self.execute_sql(sql, params, commit=commit) gcworker stdout | File "/quay-registry/data/database.py", line 240, in execute_sql gcworker stdout | self.close() gcworker stdout | File "/quay-registry/data/database.py", line 397, in close gcworker stdout | ret = super(ObservableDatabase, self).close() gcworker stdout | File "/app/lib/python3.9/site-packages/peewee.py", line 3011, in close gcworker stdout | raise OperationalError('Attempting to close database while ' gcworker stdout | peewee.OperationalError: Attempting to close database while transaction is open.
It happens on multiple workers, not just gcworker. As said, the cause seems to be incompatible operations being done at the same time on the database which cause SQLite to lock the database to ensure consistent data:
https://www2.sqlite.org/cvstrac/wiki?p=DatabaseIsLocked
This impacts all OMR installations so setting to critical priority.
Thanks!
- depends on
-
PROJQUAY-9362 Add pragma statements to sqlite driver
-
- Testing
-
- links to
- mentioned on
(3 links to, 2 mentioned on)