Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: quay-v3.8.0
Affects Version/s: quay-v3.7.0
Component/s: config-tool, quay
Labels:
- quay
- triaged

Blocked:
False
Blocked Reason:
None
Ready:
False
Product:
Quay Enterprise
Git Pull Request:
https://github.com/quay/config-tool/pull/181

Target Version:

quay-v3.7.8, quay-v3.8.0

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Client is attempting to connect to the MySQL database that is secured by TLS. When they try to validate the configuration, after uploading the database.pem file (MySQL CA), they get an error saying TLS handshake error. The config tool logs show the following:

config-editor stdout | time="2022-07-29T10:30:10Z" level=debug msg="Validating Database"
config-editor stdout | time="2022-07-29T10:30:10Z" level=debug msg="Scheme: mysql+pymysql"
config-editor stdout | time="2022-07-29T10:30:10Z" level=debug msg="Host: mpcrdcmysqldb01.xxxxxxx:3306"
config-editor stdout | time="2022-07-29T10:30:10Z" level=debug msg="Db: quay"
config-editor stdout | time="2022-07-29T10:30:10Z" level=debug msg="Params: "
config-editor stdout | time="2022-07-29T10:30:10Z" level=debug msg="CA Cert provided"
config-editor stdout | time="2022-07-29T10:30:10Z" level=debug msg="Created tls config for database successfully"
config-editor stdout | time="2022-07-29T10:30:10Z" level=debug msg="Including params tls=custom-tls"
config-editor stdout | time="2022-07-29T10:30:10Z" level=debug msg="Pinging database at quayuser:xxxxxx@tcp(mpcrdcmysqldb01.xxxxx:3306)/quay?tls=custom-tls dsn:"

There doesn't seem to be any errors reported here but the error is persistent in the UI, it's always TLS handshake error. We then tried to deploy Quay via the operator and include all db settings in the custom config bundle. We set the following:

DB_CONNECTION_ARGS: 
  ssl: 
    ca: /conf/stack/extra_ca_certs/extra_ca_cert_database.crt

We also added the extra_ca_cert_database.crt to the config bundle and attempted to deploy Quay from it. After we try that, Quay pods seem to come online but we see the following errors:

+------------------------+-------------------------------------------------------------------------------------------------------------------+--------+
| Database               | Could not connect to database. Error: Could not find database.pem in config bundle                                |   |
+------------------------+-------------------------------------------------------------------------------------------------------------------+--------+
...
gunicorn-web stdout | 2022-07-29 09:01:12,734 [239] [INFO] [data.database] Connection pooling disabled for mysql+pymysql
gunicorn-web stdout | 2022-07-29 09:01:12,745 [239] [WARNING] [health.healthcheck] [FAILED HEALTH CHECK] {'services_expanded': {'registry_gunicorn': {'status': True}, 'web_gunicorn': {'status': True}, 'service_key': {'status': True}, 'disk_space': {'status': True}, 'database': {'status': False, 'failure': 'Could not connect to the database: (1045, "Access denied for user \'quayuser\'@\'mpcrdcocpqua03xxxxxxxx\' (using password: YES)")'}, 'auth': {'status': True}}, 'notes': [], 'is_testing': False, 'config_provider': 'k8s', 'local_service_key_id': '_qsAOUj_X55KJhZ4Cw--Hxq-be1q7_pCyvSl3ybw3po', 'hostname': 'quay-registry-quay-app-b9c495776-gxkjz'}

The first error is expected, as the database.pem is not in the `/conf/stack` directory where it needs to be. However, the health check failing is not expected and is strange. We see Quay starting normally and actually doing some database operations. For instance:

exportactionlogsworker stdout | 2022-07-29 09:01:12,418 [69] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT `t1`.`queue_name` FROM `queueitem` AS `t1` WHERE (((`t1`.`available` = %s) AND (`t1`.`processing_expires` > %s)) AND (`t1`.`queue_name` LIKE %s))) AS `_wrapped`', [False, datetime.datetime(2022, 7, 29, 9, 1, 12, 417966), 'exportactionlogs/%'])

In fact, all workers start and all workers seem to have database connectivity, but health check is always failing. Furthermore, the Quay upgrade pod, which should set up the database schema, succeeds, the schema is fully set with the settings we provide:

   __   __
  /  \ /  \     ______   _    _     __   __   __
 / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / /
/ /  / /  \ \  | |  | | | |  | |  / /\ \  \   /
\ \  \ \  / /  | |__| | | |__| | / ____ \  | |
 \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|
  \__/ \__/      \ \__
                  \___\ by Red Hat
 Build, Store, and Distribute your Containers

Startup timestamp: 
Fri Jul 29 08:51:48 UTC 2022

Entering migration mode to version: head
/quay-registry/data/secscan_model/__init__.py:28: DeprecationWarning: Call to deprecated class V2SecurityScanner. (Will be replaced by a V4 API security scanner soon)
  self._legacy_model = V2SecurityScanner(app, instance_keys, storage)
Failed to validate security scanner V2 configuration
/quay-registry/data/secscan_model/__init__.py:30: DeprecationWarning: Call to deprecated class NoopV2SecurityScanner. (Will be replaced by a V4 API security scanner soon)
  self._legacy_model = NoopV2SecurityScanner()
08:51:50 INFO  [alembic.runtime.migration] Context impl MySQLImpl.
08:51:50 INFO  [alembic.runtime.migration] Will assume non-transactional DDL.

This means that the database is fully set up and Quay should bootstrap eventually. I am unsure why health checks are failing and why is Quay config tool showing a TLS handshake error. Can you please check and confirm?