Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-261

quay on OpenShift: CrashLoopBackOff after DB error

XMLWordPrintable

      Quay installed in OpenShift 4.2.13 (bare metal installation). After some days of operation, quay pod turns into status CrashLoopBackOff:

      $ oc get pods -n quay-enterprise
      NAME READY STATUS RESTARTS AGE
      example-quayecosystem-clair-659cf98799-q7gsn 1/1 Running 0 2d8h
      example-quayecosystem-clair-postgresql-c9f579b8b-bc8cv 1/1 Running 0 2d8h
      example-quayecosystem-quay-55df5876f9-m86r8 0/1 CrashLoopBackOff 642 2d8h
      example-quayecosystem-quay-config-58df9b96fc-2bxxz 1/1 Running 0 2d8h
      example-quayecosystem-quay-postgresql-74db46f585-xbvqh 1/1 Running 0 2d8h
      example-quayecosystem-redis-766b98b799-ssdm8 1/1 Running 0 2d8h
      quay-operator-7fb8f8d7c5-m6rnc 1/1 Running 0 2d8h

      $oc logs example-quayecosystem-quay-55df5876f9-m86r8 -n quay-enterprise
      (QUAY LOGO)
      Running all default registry services
      Running init script '/quay-registry/conf/init/02_get_kube_certs.sh'
      Running on kubernetes, attempting to retrieve extra certs from secret
      /opt/rh/python27/root/usr/lib/python2.7/site-packages/urllib3/connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
      InsecureRequestWarning)
      Running init script '/quay-registry/conf/init/certs_create.sh'
      Generating a 4096 bit RSA private key
      .....................................................++
      ............................................................................................++
      writing new private key to 'mitm-key.pem'


      Running init script '/quay-registry/conf/init/certs_install.sh'
      Installing extra certificates found in /conf/stack/extra_ca_certs directory
      Running init script '/quay-registry/conf/init/copy_config_files.sh'
      Running init script '/quay-registry/conf/init/nginx_conf_create.sh'
      /quay-registry/conf/init/nginx_conf_create.py:130: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
      config = yaml.load(f)
      Running init script '/quay-registry/conf/init/runmigration.sh'
      No handlers could be found for logger "util.config.provider.baseprovider"
      No handlers could be found for logger "util.config.provider.baseprovider"
      09:41:12 INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
      09:41:12 INFO [alembic.runtime.migration] Will assume non-transactional DDL.
      Running init script '/quay-registry/conf/init/supervisord_conf_create.sh'
      Running init script '/quay-registry/conf/init/zz_boot.sh'
      No handlers could be found for logger "util.config.provider.baseprovider"
      Traceback (most recent call last):
      File "./boot.py", line 136, in <module>
      main()
      File "./boot.py", line 128, in main
      setup_jwt_proxy()
      File "./boot.py", line 90, in setup_jwt_proxy
      app.config["INSTANCE_SERVICE_KEY_SERVICE"], get_audience(), expiration_date=expiration
      File "/quay-registry/util/generatepresharedkey.py", line 35, in generate_key
      logs_model.log_action("service_key_create", metadata=key_log_metadata)
      File "/quay-registry/data/logs_model/table_logs_model.py", line 236, in log_action
      timestamp=timestamp,
      File "/quay-registry/data/model/log.py", line 245, in log_action
      LogEntry3.create(**log_data)
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 6163, in create
      inst.save(force_insert=True)
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 6357, in save
      pk = self.insert(**field_dict).execute()
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 1812, in inner
      return method(self, database, *args, **kwargs)
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 1883, in execute
      return self._execute(database)
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2623, in _execute
      return super(Insert, self)._execute(database)
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2358, in _execute
      cursor = self.execute_returning(database)
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2365, in execute_returning
      cursor = database.execute(self)
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 3018, in execute
      return self.execute_sql(sql, params, commit=commit)
      File "/quay-registry/data/database.py", line 195, in execute_sql
      cursor = super(RetryOperationalError, self).execute_sql(sql, params, commit)
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 3012, in execute_sql
      self.commit()
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 2783, in _exit_
      reraise(new_type, new_type(*exc_args), traceback)
      File "/opt/rh/python27/root/usr/lib/python2.7/site-packages/peewee.py", line 3005, in execute_sql
      cursor.execute(sql, params or ())
      peewee.IntegrityError: null value in column "account_id" violates not-null constraint
      DETAIL: Failing row contains (52, 48, null, null, null, 2020-02-05 09:41:15.765335, null, {"preshared": true, "name": "https://quay.apps.ocp1.eur.ad.sag:4...).

      After this, quay must be deployed from scratch. But it never stays up longer than a week.

              Unassigned Unassigned
              b_nebel Balthasar Nebel (Inactive)
              Dongbo Yan Dongbo Yan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: