Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-10110

Documentation: Pushgateway does not start on Quay 3.16.0

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • None
    • quay-v3.16.0
    • documentation

      From the logs:

      2025-12-20 18:01:30,950 INFO spawnerr: command at '/usr/local/bin/pushgateway' is a directory
      

      This causes the following exception to occur for all workers:

      gunicorn-registry stdout | 2025-12-20 18:05:35,953 [304] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localh
      ost:9091 with grouping key {'host': 'd2a67b35de6e', 'process_name': 'registry:application', 'pid': '304'}
      ...
      gunicorn-registry stdout |   File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect
      gunicorn-registry stdout |     self._internal_connect(address)
      gunicorn-registry stdout |   File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect
      gunicorn-registry stdout |     raise _SocketError(err, strerror(err))
      gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused
      gunicorn-registry stdout | During handling of the above exception, another exception occurred:
      gunicorn-registry stdout | Traceback (most recent call last):
      gunicorn-registry stdout |   File "/quay-registry/util/metrics/prometheus.py", line 140, in run
      ...
      gunicorn-registry stdout |   File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open
      gunicorn-registry stdout |     return self.do_open(http.client.HTTPConnection, req)
      gunicorn-registry stdout |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      gunicorn-registry stdout |   File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open
      gunicorn-registry stdout |     raise URLError(err)
      gunicorn-registry stdout | urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
      

      Now Prometheus metrics are effectively not available at all since there is no pushgateway server that would expose them:

      root@laptop:~# docker exec -it quay curl https://localhost:9091/metrics -v
      *   Trying ::1:9091...
      * connect to ::1 port 9091 failed: Connection refused
      *   Trying 127.0.0.1:9091...
      * connect to 127.0.0.1 port 9091 failed: Connection refused
      * Failed to connect to localhost port 9091: Connection refused
      * Closing connection 0
      curl: (7) Failed to connect to localhost port 9091: Connection refused
      

              rhn-support-stevsmit Steven Smith
              rhn-support-ibazulic Ivan Bazulic
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: