Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-8758

Quay SQLite database locked

XMLWordPrintable

    • Incidents & Support
    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • Quay Enterprise

      When Quay is trying to do incompatible operations on the SQLite database on the same tables at the same time (for instance, SELECT/DELETE), the database can end up being locked. This is shown in the logs as a Peewee exception such as this one:

      gcworker stdout | sqlite3.OperationalError: database is locked
      gcworker stdout | During handling of the above exception, another exception occurred:
      gcworker stdout | Traceback (most recent call last):
      gcworker stdout |   File "/quay-registry/data/database.py", line 228, in execute_sql
      gcworker stdout |     cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit)
      gcworker stdout |   File "/app/lib/python3.9/site-packages/peewee.py", line 3064, in execute_sql
      gcworker stdout |     self.commit()
      gcworker stdout |   File "/app/lib/python3.9/site-packages/peewee.py", line 2831, in __exit__
      gcworker stdout |     reraise(new_type, new_type(exc_value, *exc_args), traceback)
      gcworker stdout |   File "/app/lib/python3.9/site-packages/peewee.py", line 183, in reraise
      gcworker stdout |     raise value.with_traceback(tb)
      gcworker stdout |   File "/app/lib/python3.9/site-packages/peewee.py", line 3057, in execute_sql
      gcworker stdout |     cursor.execute(sql, params or ())
      gcworker stdout | peewee.OperationalError: database is locked
      gcworker stdout | During handling of the above exception, another exception occurred:
      gcworker stdout | Traceback (most recent call last):
      gcworker stdout |   File "/quay-registry/workers/worker.py", line 86, in _operation_func
      gcworker stdout |     return operation_func()
      gcworker stdout |   File "/quay-registry/workers/gc/gcworker.py", line 77, in _garbage_collection_repos
      gcworker stdout |     garbage_collect_repo(repository)
      gcworker stdout |   File "/quay-registry/data/model/gc.py", line 248, in garbage_collect_repo
      gcworker stdout |     _purge_oci_tag(tag, context)
      gcworker stdout |   File "/quay-registry/data/model/gc.py", line 319, in _purge_oci_tag
      gcworker stdout |     delete_tag_notifications_for_tag(tag)
      gcworker stdout |   File "/quay-registry/data/model/notification.py", line 287, in delete_tag_notifications_for_tag
      gcworker stdout |     resp = TagNotificationSuccess.delete().where(TagNotificationSuccess.tag == tag.id).execute()
      gcworker stdout |   File "/app/lib/python3.9/site-packages/peewee.py", line 1845, in inner
      gcworker stdout |     return method(self, database, *args, **kwargs)
      gcworker stdout |   File "/app/lib/python3.9/site-packages/peewee.py", line 1916, in execute
      gcworker stdout |     return self._execute(database)
      gcworker stdout |   File "/app/lib/python3.9/site-packages/peewee.py", line 2400, in _execute
      gcworker stdout |     cursor = database.execute(self)
      gcworker stdout |   File "/app/lib/python3.9/site-packages/peewee.py", line 3070, in execute
      gcworker stdout |     return self.execute_sql(sql, params, commit=commit)
      gcworker stdout |   File "/quay-registry/data/database.py", line 240, in execute_sql
      gcworker stdout |     self.close()
      gcworker stdout |   File "/quay-registry/data/database.py", line 397, in close
      gcworker stdout |     ret = super(ObservableDatabase, self).close()
      gcworker stdout |   File "/app/lib/python3.9/site-packages/peewee.py", line 3011, in close
      gcworker stdout |     raise OperationalError('Attempting to close database while '
      gcworker stdout | peewee.OperationalError: Attempting to close database while transaction is open.
      

      It happens on multiple workers, not just gcworker. As said, the cause seems to be incompatible operations being done at the same time on the database which cause SQLite to lock the database to ensure consistent data:

      https://www2.sqlite.org/cvstrac/wiki?p=DatabaseIsLocked

      This impacts all OMR installations so setting to critical priority.
      Thanks!

              hgovinda Harish Govindarajulu
              rhn-support-ibazulic Ivan Bazulic
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:

                  Estimated:
                  Original Estimate - 1 week
                  1w
                  Remaining:
                  Remaining Estimate - 1 week
                  1w
                  Logged:
                  Time Spent - Not Specified
                  Not Specified