-
Bug
-
Resolution: Done
-
Blocker
-
None
-
quay-v3.16.0
From the logs:
2025-12-20 18:01:30,950 INFO spawnerr: command at '/usr/local/bin/pushgateway' is a directory
This causes the following exception to occur for all workers:
gunicorn-registry stdout | 2025-12-20 18:05:35,953 [304] [ERROR] [util.metrics.prometheus] failed to push registry to pushgateway at http://localh ost:9091 with grouping key {'host': 'd2a67b35de6e', 'process_name': 'registry:application', 'pid': '304'} ... gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 586, in connect gunicorn-registry stdout | self._internal_connect(address) gunicorn-registry stdout | File "/opt/app-root/lib64/python3.12/site-packages/gevent/_socketcommon.py", line 630, in _internal_connect gunicorn-registry stdout | raise _SocketError(err, strerror(err)) gunicorn-registry stdout | ConnectionRefusedError: [Errno 111] Connection refused gunicorn-registry stdout | During handling of the above exception, another exception occurred: gunicorn-registry stdout | Traceback (most recent call last): gunicorn-registry stdout | File "/quay-registry/util/metrics/prometheus.py", line 140, in run ... gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1373, in http_open gunicorn-registry stdout | return self.do_open(http.client.HTTPConnection, req) gunicorn-registry stdout | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ gunicorn-registry stdout | File "/usr/lib64/python3.12/urllib/request.py", line 1347, in do_open gunicorn-registry stdout | raise URLError(err) gunicorn-registry stdout | urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
Now Prometheus metrics are effectively not available at all since there is no pushgateway server that would expose them:
root@laptop:~# docker exec -it quay curl https://localhost:9091/metrics -v
* Trying ::1:9091...
* connect to ::1 port 9091 failed: Connection refused
* Trying 127.0.0.1:9091...
* connect to 127.0.0.1 port 9091 failed: Connection refused
* Failed to connect to localhost port 9091: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 9091: Connection refused
- clones
-
PROJQUAY-10083 Pushgateway does not start on Quay 3.16.0
-
- Testing
-