-
Bug
-
Resolution: Done
-
Major
-
quay-v3.6.0
Description:
This is an issue found when use Quay 3.6.0 config editor to change configurations, sometimes when Operator start to reconcile the change, Clair APP POD was failed to rolling update, get error message "server error (FATAL: sorry, too many clients already (SQLSTATE 53300))","time":"2021-10-09T06:17:58Z","message":"fatal error"", checked the max connections of quay postgersql db and Clair postgesql db, seems like it's required to increase the max connections for Clair Postgresql db, pls review this issue.
Note: Quay image is "quay-operator-bundle-container-v3.6.0-47"
oc get pod
NAME READY STATUS RESTARTS AGE
quayv360-clair-app-55858bd7c8-blrkv 0/1 CrashLoopBackOff 4 2m22s
quayv360-clair-app-55858bd7c8-mh6tm 0/1 CrashLoopBackOff 4 2m22s
quayv360-clair-app-55858bd7c8-skm94 1/1 Running 2 12m
quayv360-clair-app-55858bd7c8-xjt97 1/1 Running 2 13m
quayv360-clair-postgres-78f8ff5799-8wtqg 1/1 Running 0 9m52s
quayv360-quay-app-586bf6d878-7p6gk 1/1 Running 2 12m
quayv360-quay-app-586bf6d878-kln28 1/1 Running 2 13m
quayv360-quay-app-upgrade-g2cdc 0/1 ContainerCreating 0 5s
quayv360-quay-config-editor-7fc8ccf7f4-vtbxg 1/1 Running 0 13m
quayv360-quay-database-7bb7b4f6ff-bg9zf 1/1 Running 0 12m
quayv360-quay-mirror-844b758c77-pqtcg 1/1 Running 0 12m
quayv360-quay-mirror-844b758c77-tbkjv 1/1 Running 0 12m
quayv360-quay-postgres-init-x6qwb 0/1 Completed 0 7s
quayv360-quay-redis-54659f4454-9ctm7 1/1 Running 0 13m
oc logs quayv360-clair-app-55858bd7c8-blrkv
{"level":"info","component":"main","version":"v4.3.0","time":"2021-10-09T06:17:58Z","message":"starting"}
{"level":"info","component":"main","version":"v4.3.0","time":"2021-10-09T06:17:58Z","message":"ready"}
{"level":"info","component":"main","time":"2021-10-09T06:17:58Z","message":"launching introspection server"}
{"level":"info","component":"main","time":"2021-10-09T06:17:58Z","message":"launching http transport"}
{"level":"info","component":"introspection/New","address":":8089","time":"2021-10-09T06:17:58Z","message":"no introspection address provided; using default"}
{"level":"info","component":"initialize/Services","time":"2021-10-09T06:17:58Z","message":"begin service initialization"}
{"level":"warn","component":"introspection/New","time":"2021-10-09T06:17:58Z","message":"no health check configured; unconditionally reporting OK"}
{"level":"info","component":"introspection/Server.withPrometheus","endpoint":"/metrics","server":":8089","time":"2021-10-09T06:17:58Z","message":"configuring prometheus"}
{"level":"info","component":"introspection/New","time":"2021-10-09T06:17:58Z","message":"no distributed tracing enabled"}
{"level":"info","component":"main","time":"2021-10-09T06:17:58Z","message":"registered signal handler"}
{"level":"info","component":"initialize/Services","time":"2021-10-09T06:17:58Z","message":"end service initialization"}
{"level":"error","component":"main","error":"service initialization failed: failed to initialize indexer: failed to create ConnPool: failed to connect to `host=quayv360-clair-postgres user=postgres database=postgres`: server error (FATAL: sorry, too many clients already (SQLSTATE 53300))","time":"2021-10-09T06:17:58Z","message":"fatal error"}
Managed Quay Postgresql Database POD:
oc rsh demo36-quay-database-77bf89c77f-q557j sh-4.4$ psql psql (10.17) Type "help" for help. postgres=# SHOW max_connections; max_connections ----------------- 2000 (1 row)
Managed Clair Postgresql Database POD:
oc rsh demo36-clair-postgres-84b64f46b7-z77hc sh-4.4$ psql psql (10.17) Type "help" for help. postgres=# SHOW max_connections; max_connections ----------------- 100 (1 row)
- is related to
-
PROJQUAY-2739 Increase the amount of maximum database connections to at least 1000
-
- Closed
-
- mentioned on
