Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-2659

Quay 3.6.0 Clair APP POD was failed to rolling update caused by PSQL error "FATAL: sorry, too many clients already"

    XMLWordPrintable

Details

    • 0

    Description

      Description:

      This is an issue found when use Quay 3.6.0 config editor to change configurations, sometimes when Operator start to reconcile the change, Clair APP POD was failed to rolling update, get error message "server error (FATAL: sorry, too many clients already (SQLSTATE 53300))","time":"2021-10-09T06:17:58Z","message":"fatal error"", checked the max connections of quay postgersql db and Clair postgesql db, seems like it's required to increase the max connections for Clair Postgresql db, pls review this issue.

      Note: Quay image is "quay-operator-bundle-container-v3.6.0-47"

      oc get pod
      NAME                                           READY   STATUS              RESTARTS   AGE
      quayv360-clair-app-55858bd7c8-blrkv            0/1     CrashLoopBackOff    4          2m22s
      quayv360-clair-app-55858bd7c8-mh6tm            0/1     CrashLoopBackOff    4          2m22s
      quayv360-clair-app-55858bd7c8-skm94            1/1     Running             2          12m
      quayv360-clair-app-55858bd7c8-xjt97            1/1     Running             2          13m
      quayv360-clair-postgres-78f8ff5799-8wtqg       1/1     Running             0          9m52s
      quayv360-quay-app-586bf6d878-7p6gk             1/1     Running             2          12m
      quayv360-quay-app-586bf6d878-kln28             1/1     Running             2          13m
      quayv360-quay-app-upgrade-g2cdc                0/1     ContainerCreating   0          5s
      quayv360-quay-config-editor-7fc8ccf7f4-vtbxg   1/1     Running             0          13m
      quayv360-quay-database-7bb7b4f6ff-bg9zf        1/1     Running             0          12m
      quayv360-quay-mirror-844b758c77-pqtcg          1/1     Running             0          12m
      quayv360-quay-mirror-844b758c77-tbkjv          1/1     Running             0          12m
      quayv360-quay-postgres-init-x6qwb              0/1     Completed           0          7s
      quayv360-quay-redis-54659f4454-9ctm7           1/1     Running             0          13m
      
      
      oc logs quayv360-clair-app-55858bd7c8-blrkv
      {"level":"info","component":"main","version":"v4.3.0","time":"2021-10-09T06:17:58Z","message":"starting"}
      {"level":"info","component":"main","version":"v4.3.0","time":"2021-10-09T06:17:58Z","message":"ready"}
      {"level":"info","component":"main","time":"2021-10-09T06:17:58Z","message":"launching introspection server"}
      {"level":"info","component":"main","time":"2021-10-09T06:17:58Z","message":"launching http transport"}
      {"level":"info","component":"introspection/New","address":":8089","time":"2021-10-09T06:17:58Z","message":"no introspection address provided; using default"}
      {"level":"info","component":"initialize/Services","time":"2021-10-09T06:17:58Z","message":"begin service initialization"}
      {"level":"warn","component":"introspection/New","time":"2021-10-09T06:17:58Z","message":"no health check configured; unconditionally reporting OK"}
      {"level":"info","component":"introspection/Server.withPrometheus","endpoint":"/metrics","server":":8089","time":"2021-10-09T06:17:58Z","message":"configuring prometheus"}
      {"level":"info","component":"introspection/New","time":"2021-10-09T06:17:58Z","message":"no distributed tracing enabled"}
      {"level":"info","component":"main","time":"2021-10-09T06:17:58Z","message":"registered signal handler"}
      {"level":"info","component":"initialize/Services","time":"2021-10-09T06:17:58Z","message":"end service initialization"}
      {"level":"error","component":"main","error":"service initialization failed: failed to initialize indexer: failed to create ConnPool: failed to connect to `host=quayv360-clair-postgres user=postgres database=postgres`: server error (FATAL: sorry, too many clients already (SQLSTATE 53300))","time":"2021-10-09T06:17:58Z","message":"fatal error"}
      

      Managed Quay Postgresql Database POD:

      oc rsh demo36-quay-database-77bf89c77f-q557j
      sh-4.4$ psql
      psql (10.17)
      Type "help" for help.
      
      
      postgres=# SHOW max_connections;
       max_connections 
      -----------------
       2000
      (1 row)
      

      Managed Clair Postgresql Database POD:

      oc rsh demo36-clair-postgres-84b64f46b7-z77hc
      sh-4.4$ psql
      psql (10.17)
      Type "help" for help.
      
      
      postgres=# SHOW max_connections;
       max_connections 
      -----------------
       100
      (1 row)
      

      Attachments

        Activity

          People

            rmarasch@redhat.com Ricardo Maraschini (Inactive)
            lzha1981 luffy zhang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: