-
Bug
-
Resolution: Done
-
Major
-
None
-
None
-
False
-
False
-
Undefined
-
Hello,
I am facing big problems with Project Quay and the communication with postgres.
This is basically a plain Quay installation with a Quay(:latest/f44d76c8a0c5), a postgres and a redis container.
After some time, basically the next day after restarting the container, Quay starts throwing errors regarding the DB connection. A restart of the quay container fixes it, until the next day.
I am using a default Redis container and a mainly default postgres container (only set max_connections to 1000 in hope it will help conquer the problem).
Expected behaviour: Quay should work longer than a day
Actual behaviour: It doesn't
Log:
quay_1 | gunicorn-web stdout | 2020-10-28 08:51:51,991 [307] [ERROR] [gunicorn.error] Error handling request /favicon.ico/ quay_1 | gunicorn-web stdout | Traceback (most recent call last): quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2993, in connect quay_1 | gunicorn-web stdout | self._state.set_connection(self._connect()) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3688, in _connect quay_1 | gunicorn-web stdout | conn = psycopg2.connect(database=self.database, **self.connect_params) quay_1 | gunicorn-web stdout | File "/usr/local/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect quay_1 | gunicorn-web stdout | conn = _connect(dsn, connection_factory=connection_factory, **kwasync) quay_1 | gunicorn-web stdout | psycopg2.OperationalError: FATAL: sorry, too many clients already quay_1 | gunicorn-web stdout | During handling of the above exception, another exception occurred: quay_1 | gunicorn-web stdout | Traceback (most recent call last): quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3055, in execute_sql quay_1 | gunicorn-web stdout | cursor = self.cursor(commit) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3039, in cursor quay_1 | gunicorn-web stdout | self.connect() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2996, in connect quay_1 | gunicorn-web stdout | self._initialize_connection(self._state.conn) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2831, in __exit__ quay_1 | gunicorn-web stdout | reraise(new_type, new_type(exc_value, *exc_args), traceback) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 183, in reraise quay_1 | gunicorn-web stdout | raise value.with_traceback(tb) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2993, in connect quay_1 | gunicorn-web stdout | self._state.set_connection(self._connect()) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3688, in _connect quay_1 | gunicorn-web stdout | conn = psycopg2.connect(database=self.database, **self.connect_params) quay_1 | gunicorn-web stdout | File "/usr/local/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect quay_1 | gunicorn-web stdout | conn = _connect(dsn, connection_factory=connection_factory, **kwasync) quay_1 | gunicorn-web stdout | peewee.OperationalError: FATAL: sorry, too many clients already quay_1 | gunicorn-web stdout | During handling of the above exception, another exception occurred: quay_1 | gunicorn-web stdout | Traceback (most recent call last): quay_1 | gunicorn-web stdout | File "/quay-registry/data/database.py", line 213, in execute_sql quay_1 | gunicorn-web stdout | cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3064, in execute_sql quay_1 | gunicorn-web stdout | self.commit() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2831, in __exit__ quay_1 | gunicorn-web stdout | reraise(new_type, new_type(exc_value, *exc_args), traceback) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 183, in reraise quay_1 | gunicorn-web stdout | raise value.with_traceback(tb) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3055, in execute_sql quay_1 | gunicorn-web stdout | cursor = self.cursor(commit) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3039, in cursor quay_1 | gunicorn-web stdout | self.connect() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2996, in connect quay_1 | gunicorn-web stdout | self._initialize_connection(self._state.conn) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2831, in __exit__ quay_1 | gunicorn-web stdout | reraise(new_type, new_type(exc_value, *exc_args), traceback) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 183, in reraise quay_1 | gunicorn-web stdout | raise value.with_traceback(tb) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2993, in connect quay_1 | gunicorn-web stdout | self._state.set_connection(self._connect()) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3688, in _connect quay_1 | gunicorn-web stdout | conn = psycopg2.connect(database=self.database, **self.connect_params) quay_1 | gunicorn-web stdout | File "/usr/local/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect quay_1 | gunicorn-web stdout | conn = _connect(dsn, connection_factory=connection_factory, **kwasync) quay_1 | gunicorn-web stdout | peewee.OperationalError: FATAL: sorry, too many clients already quay_1 | gunicorn-web stdout | During handling of the above exception, another exception occurred: quay_1 | gunicorn-web stdout | Traceback (most recent call last): quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2993, in connect quay_1 | gunicorn-web stdout | self._state.set_connection(self._connect()) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3688, in _connect quay_1 | gunicorn-web stdout | conn = psycopg2.connect(database=self.database, **self.connect_params) quay_1 | gunicorn-web stdout | File "/usr/local/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect quay_1 | gunicorn-web stdout | conn = _connect(dsn, connection_factory=connection_factory, **kwasync) quay_1 | gunicorn-web stdout | psycopg2.OperationalError: FATAL: sorry, too many clients already quay_1 | gunicorn-web stdout | During handling of the above exception, another exception occurred: quay_1 | gunicorn-web stdout | Traceback (most recent call last): quay_1 | gunicorn-web stdout | File "/quay-registry/data/database.py", line 219, in execute_sql quay_1 | gunicorn-web stdout | cursor = self.cursor() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3039, in cursor quay_1 | gunicorn-web stdout | self.connect() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2996, in connect quay_1 | gunicorn-web stdout | self._initialize_connection(self._state.conn) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2831, in __exit__ quay_1 | gunicorn-web stdout | reraise(new_type, new_type(exc_value, *exc_args), traceback) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 183, in reraise quay_1 | gunicorn-web stdout | raise value.with_traceback(tb) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2993, in connect quay_1 | gunicorn-web stdout | self._state.set_connection(self._connect()) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3688, in _connect quay_1 | gunicorn-web stdout | conn = psycopg2.connect(database=self.database, **self.connect_params) quay_1 | gunicorn-web stdout | File "/usr/local/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect quay_1 | gunicorn-web stdout | conn = _connect(dsn, connection_factory=connection_factory, **kwasync) quay_1 | gunicorn-web stdout | peewee.OperationalError: FATAL: sorry, too many clients already quay_1 | gunicorn-web stdout | During handling of the above exception, another exception occurred: quay_1 | gunicorn-web stdout | Traceback (most recent call last): quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 55, in handle quay_1 | gunicorn-web stdout | self.handle_request(listener_name, req, client, addr) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request quay_1 | gunicorn-web stdout | super().handle_request(listener_name, req, sock, addr) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request quay_1 | gunicorn-web stdout | respiter = self.wsgi(environ, resp.start_response) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2463, in __call__ quay_1 | gunicorn-web stdout | return self.wsgi_app(environ, start_response) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__ quay_1 | gunicorn-web stdout | return self.app(environ, start_response) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2449, in wsgi_app quay_1 | gunicorn-web stdout | response = self.handle_exception(e) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 269, in error_router quay_1 | gunicorn-web stdout | return original_handler(e) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1866, in handle_exception quay_1 | gunicorn-web stdout | reraise(exc_type, exc_value, tb) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise quay_1 | gunicorn-web stdout | raise value quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app quay_1 | gunicorn-web stdout | response = self.full_dispatch_request() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request quay_1 | gunicorn-web stdout | rv = self.handle_user_exception(e) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 269, in error_router quay_1 | gunicorn-web stdout | return original_handler(e) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception quay_1 | gunicorn-web stdout | reraise(exc_type, exc_value, tb) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise quay_1 | gunicorn-web stdout | raise value quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request quay_1 | gunicorn-web stdout | rv = self.dispatch_request() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request quay_1 | gunicorn-web stdout | return self.view_functions[rule.endpoint](**req.view_args) quay_1 | gunicorn-web stdout | File "/quay-registry/util/cache.py", line 36, in add_no_cache quay_1 | gunicorn-web stdout | response = f(*args, **kwargs) quay_1 | gunicorn-web stdout | File "/quay-registry/auth/decorators.py", line 40, in wrapper quay_1 | gunicorn-web stdout | result = handler(auth_header) quay_1 | gunicorn-web stdout | File "/quay-registry/auth/cookie.py", line 26, in validate_session_cookie quay_1 | gunicorn-web stdout | db_user = current_user.db_user() quay_1 | gunicorn-web stdout | File "/quay-registry/data/model/user.py", line 1374, in db_user quay_1 | gunicorn-web stdout | self._db_user = get_user_by_uuid(self._uuid) quay_1 | gunicorn-web stdout | File "/quay-registry/data/model/user.py", line 845, in get_user_by_uuid quay_1 | gunicorn-web stdout | return User.get(User.uuid == user_uuid, User.organization == False) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 6318, in get quay_1 | gunicorn-web stdout | return sq.get() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 6741, in get quay_1 | gunicorn-web stdout | return clone.execute(database)[0] quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 1845, in inner quay_1 | gunicorn-web stdout | return method(self, database, *args, **kwargs) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 1916, in execute quay_1 | gunicorn-web stdout | return self._execute(database) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2087, in _execute quay_1 | gunicorn-web stdout | cursor = database.execute(self) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3070, in execute quay_1 | gunicorn-web stdout | return self.execute_sql(sql, params, commit=commit) quay_1 | gunicorn-web stdout | File "/quay-registry/data/database.py", line 222, in execute_sql quay_1 | gunicorn-web stdout | self.commit() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2831, in __exit__ quay_1 | gunicorn-web stdout | reraise(new_type, new_type(exc_value, *exc_args), traceback) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 183, in reraise quay_1 | gunicorn-web stdout | raise value.with_traceback(tb) quay_1 | gunicorn-web stdout | File "/quay-registry/data/database.py", line 219, in execute_sql quay_1 | gunicorn-web stdout | cursor = self.cursor() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3039, in cursor quay_1 | gunicorn-web stdout | self.connect() quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2996, in connect quay_1 | gunicorn-web stdout | self._initialize_connection(self._state.conn) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2831, in __exit__ quay_1 | gunicorn-web stdout | reraise(new_type, new_type(exc_value, *exc_args), traceback) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 183, in reraise quay_1 | gunicorn-web stdout | raise value.with_traceback(tb) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 2993, in connect quay_1 | gunicorn-web stdout | self._state.set_connection(self._connect()) quay_1 | gunicorn-web stdout | File "/usr/local/lib/python3.6/site-packages/peewee.py", line 3688, in _connect quay_1 | gunicorn-web stdout | conn = psycopg2.connect(database=self.database, **self.connect_params) quay_1 | gunicorn-web stdout | File "/usr/local/lib64/python3.6/site-packages/psycopg2/__init__.py", line 126, in connect quay_1 | gunicorn-web stdout | conn = _connect(dsn, connection_factory=connection_factory, **kwasync) quay_1 | gunicorn-web stdout | peewee.OperationalError: FATAL: sorry, too many clients already
