-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
quay-v3.2.0
Quay installed in OpenShift 4.2.13 (bare metal installation). After some days of operation, quay pod turns into status CrashLoopBackOff:
$ oc get pods -n quay-enterprise
NAME READY STATUS RESTARTS AGE
example-quayecosystem-clair-659cf98799-q7gsn 1/1 Running 0 2d8h
example-quayecosystem-clair-postgresql-c9f579b8b-bc8cv 1/1 Running 0 2d8h
example-quayecosystem-quay-55df5876f9-m86r8 0/1 CrashLoopBackOff 642 2d8h
example-quayecosystem-quay-config-58df9b96fc-2bxxz 1/1 Running 0 2d8h
example-quayecosystem-quay-postgresql-74db46f585-xbvqh 1/1 Running 0 2d8h
example-quayecosystem-redis-766b98b799-ssdm8 1/1 Running 0 2d8h
quay-operator-7fb8f8d7c5-m6rnc 1/1 Running 0 2d8h
$oc logs example-quayecosystem-quay-55df5876f9-m86r8 -n quay-enterprise
(QUAY LOGO)
Running all default registry services
Running init script '/quay-registry/conf/init/02_get_kube_certs.sh'
Running on kubernetes, attempting to retrieve extra certs from secret
/opt/rh/python27/root/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
Running init script '/quay-registry/conf/init/certs_create.sh'
Generating a 4096 bit RSA private key
.....................................................++
............................................................................................++
writing new private key to 'mitm-key.pem'
Running init script '/quay-registry/conf/init/certs_install.sh'
Installing extra certificates found in /conf/stack/extra_ca_certs directory
Running init script '/quay-registry/conf/init/copy_config_files.sh'
Running init script '/quay-registry/conf/init/nginx_conf_create.sh'
/quay-registry/conf/init/nginx_conf_create.py:130: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(f)
Running init script '/quay-registry/conf/init/runmigration.sh'
No handlers could be found for logger "util.config.provider.baseprovider"
No handlers could be found for logger "util.config.provider.baseprovider"
09:41:12 INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
09:41:12 INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Running init script '/quay-registry/conf/init/supervisord_conf_create.sh'
Running init script '/quay-registry/conf/init/zz_boot.sh'
No handlers could be found for logger "util.config.provider.baseprovider"
Traceback (most recent call last):
File "./boot.py", line 136, in <module>
main()
File "./boot.py", line 128, in main
setup_jwt_proxy()
File "./boot.py", line 90, in setup_jwt_proxy
app.config["INSTANCE_SERVICE_KEY_SERVICE"], get_audience(), expiration_date=expiration
File "/quay-registry/util/generatepresharedkey.py", line 35, in generate_key
logs_model.log_action("service_key_create", metadata=key_log_metadata)
File "/quay-registry/data/logs_model/table_logs_model.py", line 236, in log_action
timestamp=timestamp,
File "/quay-registry/data/model/log.py", line 245, in log_action
LogEntry3.create(**log_data)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 6163, in create
inst.save(force_insert=True)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 6357, in save
pk = self.insert(**field_dict).execute()
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 1812, in inner
return method(self, database, *args, **kwargs)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 1883, in execute
return self._execute(database)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2623, in _execute
return super(Insert, self)._execute(database)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2358, in _execute
cursor = self.execute_returning(database)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2365, in execute_returning
cursor = database.execute(self)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 3018, in execute
return self.execute_sql(sql, params, commit=commit)
File "/quay-registry/data/database.py", line 195, in execute_sql
cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 3012, in execute_sql
self.commit()
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2783, in _exit_
reraise(new_type, new_type(*exc_args), traceback)
File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 3005, in execute_sql
cursor.execute(sql, params or ())
peewee.IntegrityError: null value in column "account_id" violates not-null constraint
DETAIL: Failing row contains (52, 48, null, null, null, 2020-02-05 09:41:15.765335, null, {"preshared": true, "name": "https://quay.apps.ocp1.eur.ad.sag:4...).
After this, quay must be deployed from scratch. But it never stays up longer than a week.